Final  Report 


Real  Time  Quality  Control  Methods  for  Cued 
EMI  Data  Collection 

ESTCP  Project  MR-201264 


JANUARY  2016 


Jonathan  Miller 

White  River  Technologies,  Inc. 


Distribution  Statement  A 

This  document  has  been  cleared  for  public  release 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  0704-0188 


The  public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data 
sources,  gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other 
aspect  of  this  collection  of  information,  including  suggestions  for  reducing  the  burden,  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information 
Operations  and  Reports  (0704-0188),  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other 
provision  of  law,  no  person  shall  be  subject  to  any  penalty  for  failing  to  comply  with  a  collection  of  information  if  it  does  not  display  a  currently  valid  OMB  control  number. 

PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 


1.  REPORT  DATE  (DD-MM-YYYY) 

01/12/2016 


2.  REPORT  TYPE 

Final  Report 


3.  DATES  COVERED  (From  -  To) 

07/24/2012  -01/12/2016 


4.  TITLE  AND  SUBTITLE 

Real  Time  Quality  Control  Methods  for  Cued  EMI  Data 
Collection 


5a.  CONTRACT  NUMBER 

W91 2HQ-1 2-C-0065 


5b.  GRANT  NUMBER 


5c.  PROGRAM  ELEMENT  NUMBER 


6.  AUTHOR(S) 

Jonathan  S.  Miller,  White  River  Technologies 


5d.  PROJECT  NUMBER 

MR-201264 


5e.  TASK  NUMBER 


5f.  WORK  UNIT  NUMBER 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Jonathan  S.  Miller,  White  River  Technologies,!  242  Chestnut  Street 
Newton,  MA  02464 

Dr.  Leonard  Pasion,  Black  Tusk  Geophysics,  401  / 1755  West  Broadway 
Vancouver,  BC  Canada  V6J  4S5 


8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Environmental  Security  Technology  Certification  Program  Program  Office 
4800  Mark  Center  Drive 
Suite  17D03 

Alexandria,  VA  22350-3605 


10.  SPONSOR/MONITOR’S  ACRONYM(S) 

ESTCP 


11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  is  unlimited. 


13.  SUPPLEMENTARY  NOTES 


14.  ABSTRACT 

This  project  evaluated  the  effectiveness  of  in-field  quality  control  (QC)  procedures  during  cued 
EMI  data  collection. The  in-field  QC  approach  includes  the  use  by  cued  sensor  operators  of  a 
real-time  inversion  software  module  that  provides  immediate  output  of  features  associated  with 
each  anomaly  investigated  by  cued  EMI  data  collection. 


15.  SUBJECT  TERMS 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 

18.  NUMBER 

19a.  NAME  OF  RESPONSIBLE  PERSON 

a.  REPORT 

b.  ABSTRACT 

c.  THIS  PAGE 

ABSTRACT 

OF 

PAGES 

Jonathan  S.  Miller 

19b.  TELEPHONE  NUMBER  (Include  area  code) 

u 

u 

u 

uu 

85 

603-727-9643 

Standard  Form  298  (Rev.  8/98) 
Prescribed  by  ANSI  Std.  Z39.18 


INSTRUCTIONS  FOR  COMPLETING  SF  298 


1.  REPORT  DATE.  Full  publication  date,  including 
day,  month,  if  available.  Must  cite  at  least  the  year 
and  be  Year  2000  compliant,  e.g.  30-06-1998; 
xx-06-1998;  xx-xx-1998. 

2.  REPORT  TYPE.  State  the  type  of  report,  such  as 
final,  technical,  interim,  memorandum,  master's 
thesis,  progress,  quarterly,  research,  special,  group 
study,  etc. 

3.  DATE  COVERED.  Indicate  the  time  during 
which  the  work  was  performed  and  the  report  was 
written,  e.g.,  Jun  1997  -  Jun  1998;  1-10  Jun  1996; 
May  -  Nov  1998;  Nov  1998. 

4.  TITLE.  Enter  title  and  subtitle  with  volume 
number  and  part  number,  if  applicable.  On  classified 
documents,  enter  the  title  classification  in 
parentheses. 

5a.  CONTRACT  NUMBER.  Enter  all  contract 
numbers  as  they  appear  in  the  report,  e.g. 
F33315-86-C-5169. 

5b.  GRANT  NUMBER.  Enter  all  grant  numbers  as 
they  appear  in  the  report,  e.g.  AFOSR-82-1234. 

5c.  PROGRAM  ELEMENT  NUMBER.  Enter  all 
program  element  numbers  as  they  appear  in  the 
report,  e.g.  61 101  A. 

5e.  TASK  NUMBER.  Enter  all  task  numbers  as  they 
appear  in  the  report,  e.g.  05;  RF0330201 ;  T41 1 2. 

5f.  WORK  UNIT  NUMBER.  Enter  all  work  unit 
numbers  as  they  appear  in  the  report,  e.g.  001 ; 
AFAPL304801 05. 

6.  AUTHOR(S).  Enter  name(s)  of  person(s) 
responsible  for  writing  the  report,  performing  the 
research,  or  credited  with  the  content  of  the  report. 
The  form  of  entry  is  the  last  name,  first  name,  middle 
initial,  and  additional  qualifiers  separated  by  commas, 
e.g.  Smith,  Richard,  J,  Jr. 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND 
ADDRESS(ES).  Self-explanatory. 


8.  PERFORMING  ORGANIZATION  REPORT  NUMBER. 

Enter  all  unique  alphanumeric  report  numbers  assigned 
by  the  performing  organization,  e.g.  BRL-1234; 
AFWL-TR-85-401 7-Vol-21  -PT-2. 

9.  SPONSORING/MONITORING  AGENCY  NAME(S) 
AND  ADDRESS(ES).  Enter  the  name  and  address  of 
the  organization(s)  financially  responsible  for  and 
monitoring  the  work. 

10.  SPONSOR/MONITOR'S  ACRONYM(S).  Enter,  if 
available,  e.g.  BRL,  ARDEC,  NADC. 

11.  SPONSOR/MONITOR'S  REPORT  NUMBER(S). 

Enter  report  number  as  assigned  by  the  sponsoring/ 
monitoring  agency,  if  available,  e.g.  BRL-TR-829;  -215. 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT. 

Use  agency-mandated  availability  statements  to  indicate 
the  public  availability  or  distribution  limitations  of  the 
report.  If  additional  limitations/  restrictions  or  special 
markings  are  indicated,  follow  agency  authorization 
procedures,  e.g.  RD/FRD,  PROPIN, 

ITAR,  etc.  Include  copyright  information. 

13.  SUPPLEMENTARY  NOTES.  Enter  information 

not  included  elsewhere  such  as:  prepared  in  cooperation 
with;  translation  of;  report  supersedes;  old  edition 
number,  etc. 

14.  ABSTRACT.  A  brief  (approximately  200  words) 
factual  summary  of  the  most  significant  information. 

15.  SUBJECT  TERMS.  Key  words  or  phrases 
identifying  major  concepts  in  the  report. 

16.  SECURITY  CLASSIFICATION.  Enter  security 
classification  in  accordance  with  security  classification 
regulations,  e.g.  U,  C,  S,  etc.  If  this  form  contains 
classified  information,  stamp  classification  level  on  the 
top  and  bottom  of  this  page. 

17.  LIMITATION  OF  ABSTRACT.  This  block  must  be 
completed  to  assign  a  distribution  limitation  to  the 
abstract.  Enter  UU  (Unclassified  Unlimited)  or  SAR 
(Same  as  Report).  An  entry  in  this  block  is  necessary  if 
the  abstract  is  to  be  limited. 


Standard  Form  298  Back  (Rev.  8/98) 


FINAL  REPORT 

Real  Time  Quality  Control  Methods  for  Cued  EMI  Data 

Collection 


ESTCP  Project  MR-201264 


Jonathan  Miller 
White  River  Technologies,  Inc. 


January  2016 


EST CP -MR-201264  Real  Time  Quality  Control  Methods  for  Cued  EMI  Data  Collection 


TABLE  OF  CONTENTS 

1.  INTRODUCTION . 1 

1.1  BACKGROUND . 2 

1 .2  OBJECTIVE  OF  THE  DEMONSTRATION . 3 

1 .3  REGULATORY  DRIVERS . 4 

2.  TECHNOLOGY . 4 

2.1  TECHNOLOGY  DESCRIPTION . 4 

2.2  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGY . 9 

3.  PERFORMANCE  OBJECTIVES . 10 

3 . 1  OBJECTIVE:  IDENTIFICATION  OF  ALL  RECOLLECT  OPPORTUNITIES . 12 

3.1.1  Data  Requirements . 12 

3.1.2  Metric . 12 

3.1.3  Success  Criteria . 13 

3.2  OBJECTIVE:  EFFECTIVE  CORRECTIVE  ACTION . 13 

3.2.1  Data  Requirements . 13 

3.2.2  Metric . 13 

3.2.3  Success  Criteria . 13 

3.3  OBJECTIVE:  EFFECTIVE  IN-FIELD  CHARACTERIZATION . 13 

3.3.1  Data  Requirements . 13 

3.3.2  Metric . 13 

3.3.3  Success  Criteria . 14 

3.4  OBJECTIVE:  EFFECTIVE  QUALITY  METRICS . 14 

3.4.1  Data  Requirements . 14 

3.4.2  Metric . 14 

3.4.3  Success  Criteria . 14 

3.5  OBJECTIVE:  PRODUCTION  RATE . 14 

3.5.1  Data  Requirements . 14 

3.5.2  Metric . 15 

3.5.3  Success  Criteria . 15 

3.6  Quantifying  Improvements  in  Target  Features . 15 

4.  SITE  DESCRIPTION . 23 

5.  TEST  DESIGN . 25 

White  River  Technologies  i  January  2016 


EST CP -MR-201264  Real  Time  Quality  Control  Methods  for  Cued  EMI  Data  Collection 


6.  DATA  ANALYSIS  PLAN . 25 

6.1  PARAMETER  ESTIMATION . 25 

6.2  PERFORMANCE  ANALYSIS . 26 

6.2.1  Objective  3.1:  Identification  of  All  Recollect  Opportunities . 26 

6.2.2  Objective  3.2:  Effective  Corrective  Action . 26 

6.2.3  Objective  3.3:  Effective  In-Field  Characterization . 26 

6.2.4  Objective  3.4:  Effective  Quality  Metrics . 27 

6.2.5  Objective  3.5:  Production  Rate . 27 

7.  PERFORMANCE  ASSESSMENT . 27 

7.1  Objective  3.1:  Identification  of  All  Recollect  Opportunities . 27 

7.2  Objective  3.2:  Effective  Corrective  Action . 29 

7.3  Objective  3.3:  Effective  In-Field  Characterization . 31 

7.4  Objective  3.4:  Effective  Quality  Metrics . 32 

7.5  Objective  3.5:  Production  Rate . 36 

7.6  Qualitative  Objectives . 37 

8.  COST  ASSESSMENT . 37 

8.1  Cost  Drivers . 37 

9.  IMPLEMENTATION  ISSUES . 38 

10.  REFERENCES . 39 

Appendix  A  -  Interim  Proveout  Report . 40 

ACRONYMS 

AOI  -  Area  of  Interest 

API  -  Application  Programming  Interface 

DGM  -  Digital  Geophysical  Mapping 

DQO  -  Data  Quality  Objective 

EMI  -  Electromagnetic  Induction 

ESTCP  -  Environmental  Security  Technology  Certification  Program 

GPS  -  Global  Positioning  System 

IMU  -  Inertial  Measurement  Unit 

ISO  -  Industry  Standard  Object 

IVS  -  Instrument  Verification  Strip 

MEC  -  Munitions  and  Explosives  of  Concern 

MM  -  MetalMapper 


White  River  Technologies 


2 


January  2016 


EST CP -MR-201264  Real  Time  Quality  Control  Methods  for  Cued  EMI  Data  Collection 


MMRP  -  Military  Munitions  Response  Program 
MR  -  Munitions  Response 
MRS  -  Munitions  Response  Site 
Preac  -  Percentage  of  reacquisitions 

Prec(I)  -  Percentage  of  recollects  resulting  in  improved  classification 
QC  -  Quality  Control 
Rx  -  Receiver 

SNR  -  Signal  to  Noise  Ratio 
SOP  -  Standard  Operating  Procedure 
TOI  -  Target  of  Interest 
Tx  -  Transmitter 

UTC  -  Coordinated  Universal  Time 
UTM  -  Universal  Transverse  Mercator 
UXO  -  Unexploded  Ordnance 
WMA  -  Waikoloa  Maneuver  Area 


FIGURES 

Figure  1.  Typical  operator  display  setup  showing  QC  and  acquisition  interfaces.  The  QC  Module 
provides  the  main  navigation  and  analysis  interfaces,  while  EM3D  provides  the  acquisition 
capability  with  a  simple  acquire  button . 6 

Figure  2.  LEFT:  The  navigation  interface  displays  the  anomaly  coordinates  and  the  sensor 
location.  RIGHT :  As  the  operator  moves  the  sensor  closer  to  an  anomaly,  the  display  performs 
an  auto-zoom  to  facilitate  accurate  positioning  of  the  sensor  head  over  the  anomaly.  After  the 
data  are  acquired,  the  QC  module  updates  the  navigation  display  with  the  estimated  target 
location  (pink  dot) . 7 

Figure  3.  LEFT:  Target  libraries  are  built  from  data  collected  in  controlled  calibration  areas  within 
the  site.  This  image  shows  a  37mm  projectile  placed  in  a  test  pit  for  library  data  collection. 
RIGHT:  Library  features  are  subsequently  loaded  into  the  QC  module.  The  QC  interface  displays 
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Figure  4.  LEFT:  QC  module  performance  tests  were  conducted  by  acquiring  MetalMapper 
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Figure  5.  TOP  LEFT:  The  red  dot  shows  the  location  of  the  target  pick.  The  pink  dot  shows  the 
estimated  target  location.  TOP  CENTER:  The  sensor  is  repositioned  over  the  estimated  location, 
which  is  now  represented  by  the  red  dot.  TOP  RIGHT:  If  the  operator  determines  the  quality  of 
the  recollected  data  is  acceptable,  the  dot  turns  green  indicating  that  the  data  corresponding  to 
this  anomaly  have  met  the  quality  requirements.  BOTTOM  LEFT:  QC  interface  for  the 
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recollected  data.  BOTTOM  RIGHT:  The  actual  target  (37  mm  projectile)  and  target  pick  location 
(orange  paint) . 9 

Figure  6.  LEFT:  Polarizabilities  recovered  from  inversion  of  initial  data  acquired  with  55cm 
offset  from  target.  RIGHT :  Polarizabilities  recovered  from  inversion  of  recollect  data  acquired 
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secondary,  and  tertiary  principal  polarizabilities,  respectively  (LI,  L2,  and  L3,  respectively).  The 
grey  lines  represent  the  principal  polarizabilities  for  the  reference  library  match  (60mm 
mortar) . 16 

Figure  7.  Model  noise  values  for  LI  (top),  L2  (middle),  and  L3  (bottom).  Dashed  lines  represent 
noise  values  associated  with  the  initial  data  file;  solid  lines  represent  noise  values  associated  with 
the  recollect  data  file.  It  is  clear  that  only  the  L3  shows  any  significant  improvements  in  noise 
associated  with  the  recollect . 18 

Figure  8.  LEFT:  Polarizabilities  recovered  from  inversion  of  initial  data  acquired  with  >35cm 
offset  from  target.  RIGHT :  Polarizabilities  recovered  from  inversion  of  recollect  data  acquired 
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grey  lines  represent  the  principal  polarizabilities  for  the  reference  library  match  (37mm 
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Figure  14.  Initial  acquisition  model  fit  comparing  concatenated  data  (blue)  and  model  (red) 
outputs.  The  data  produce  a  good  model  fit,  but  the  solution  is  non-unique . 22 

Figure  15.  Recollect  model  fit  comparing  concatenated  data  (blue)  and  model  (red)  outputs.  The 
data  produce  a  good  model  fit  and  a  unique  solution . 23 

Figure  16.  Former  Waikoloa  Maneuver  Area  (from  USACE  Waikoloa  map) . 24 

Figure  17.  Former  Waikoloa  Maneuver  Area  site  photos . 24 


White  River  Technologies 


4 


January  2016 


EST CP -MR-201264  Real  Time  Quality  Control  Methods  for  Cued  EMI  Data  Collection 


Figure  18.  Independent  scoring  results  for  Dartmouth  College  WMA  MetalMapper  analysis 
indicate  two  difficult  targets:  WK-1027  and  WK-1047 . 28 

Figure  19.  Polarizabilities  obtained  from  two  different  acquisitions  over  WK-724,  a  37mm  seed. 
For  the  initial  acquisition,  the  sensor  was  offset  34  cm  from  the  target.  The  recollect  placed  the 
sensor  at  an  offset  of  13  cm  from  the  target.  The  polarizabilities  (blue,  red,  and  green  lines) 
show  a  much  better  match  to  the  library  (dark  grey  lines)  in  the  recollect . 29 

Figure  20.  Polarizabilities  obtained  from  two  different  acquisitions  over  WK-1053,  a  piece  of 
munitions  debris.  The  recollect  provides  better  constraint  on  the  third  polarizability  (green  line). 
. 30 

Figure  21.  Example  of  the  recollect  providing  better  resolution  of  target  features.  In  this  case  the 
initial  acquisition  produced  polarizabilities  dominated  by  the  ground  response.  The  recollect 
provides  features  that  more  accurately  represent  the  actual  target  (a  piece  of  frag) . 31 

Figure  22.  Polarizabilities  associated  with  anomaly  WK-68,  a  piece  of  frag.  Here  the  initial 
acquisition  shows  good  resolution  of  target  features;  however,  the  recollect  provides  features 
dominated  by  the  ground  response . 32 

Figure  23.  Polarizabilities  recovered  from  the  initial  acquisition  data  (left)  and  the  recollect  data 
(right).  These  polarizabilities  correspond  to  anomaly  WK-267,  a  medium  ISO  seed.  Here  the 
recollect  was  unnecessary . 33 

Figure  24.  Polarizabilities  recovered  from  the  initial  acquisition  data  (left)  and  the  recollect  data 
(right).  These  polarizabilities  correspond  to  anomaly  WK-169,  a  piece  of  frag.  Here  the 
recollect  was  unnecessary . 33 

Figure  25.  Polarizabilities  recovered  from  the  initial  acquisition  data  (left)  and  the  recollect  data 
(right).  These  polarizabilities  correspond  to  anomaly  WK-880,  a  fuze  clip.  The  initial  data  do 
not  fully  constrain  the  third  polarizability  (green  line)  and  indicate  that  the  recollect  was 
beneficial . 34 

Figure  26.  Possible  decision  flow  for  in-field  QC.  This  decision  flow  incorporates  the  additional 
fit  and  model  noise  metrics  along  with  the  primary  target  location  metric  to  support  the  decision  to 
recollect  or  not.  The  objective  of  this  type  of  decision  flow  would  be  to  reduce  unnecessary 
recollects  that  may  occur  within  some  margin  of  error  of  the  30  cm  offset  threshold  (e.g.,  25  -  40 
cm).  All  metrics  including  the  offset  threshold  could  be  adjusted  for  site  specific  requirements 
like  smallest  TOI,  target  density,  etc.  to  reflect  different  tolerances  for  data  recollects . 35 


TABLES 


Table  1.  Performance  Objectives . 11 

Table  2.  Feature  Quality  Metrics  for  60mm  Mortar . 16 


White  River  Technologies 


5 


January  2016 


EST CP -MR-201264  Real  Time  Quality  Control  Methods  for  Cued  EMI  Data  Collection 


Table  3.  Feature  Quality  Metrics  for  Small  ISO40 . 19 

Table  5.  Cost  Model  for  In-Field  Quality  Control  for  Cued  EMI  Surveys . 37 


ACKNOWLEDGEMENTS 

This  project  included  contributions  from  several  performers  throughout  the  initial  development 
and  proveout  stages  as  well  as  during  the  final  demonstration  of  the  technology.  Thanks  to 
Kevin  Kingdon,  Dave  Lutes,  and  Barry  Zelt  (with  Sky  Research  at  the  time,  currently  with  Black 
Tusk  Geophysics)  who  provided  assistance  during  the  initial  development  and  proveout  of  the 
in-field  QC  technology.  Thanks  also  to  Elise  Goggin  (with  USACE  Huntsville  at  the  time, 
currently  with  Tetra  Tech)  who  provided  feedback  regarding  the  utility  of  the  in-field  QC 
software  during  cued  surveys.  Mike  Sherwin  and  John  Baptiste,  both  with  Parsons,  assisted  with 
the  live  site  demonstration  of  the  technology  by  implementing  the  in-field  QC  process  during 
their  cued  MetalMapper  survey  at  the  Former  Waikoloa  Maneuver  Area.  Fridon  Shubitidze  at 
Dartmouth  College  (also  with  White  River  Technologies)  provided  final  classification  results 
from  the  Waikoloa  demonstration.  These  results  provided  a  basis  for  a  retrospective  analysis  of 
the  in-field  QC  performance. 


White  River  Technologies 


6 


January  2016 


EST CP -MR-201264  Real  Time  Quality  Control  Methods  for  Cued  EMI  Data  Collection 


Executive  Summary 

This  project  evaluated  the  effectiveness  of  in-field  quality  control  (QC)  procedures  during  cued 
EMI  data  collection.  The  in-field  QC  approach  includes  the  use  by  cued  sensor  operators  of  a 
real-time  inversion  software  module  that  provides  immediate  output  of  features  associated  with 
each  anomaly  investigated  by  cued  EMI  data  collection.  Among  the  relevant  features  provided 
by  the  software  is  an  estimate  of  the  location  of  the  buried  target.  If  the  lateral  offset  of  this 
estimated  location  is  greater  than  30  cm  from  the  center  of  the  cued  sensor,  the  sensor  operator 
can  reposition  the  sensor  over  the  estimated  source  location  and  recollect  the  cued  data.  Visual 
interpretation  of  the  sensor  location,  the  estimated  target  location,  and  other  target  features  such 
as  electromagnetic  polarizabilities  is  enabled  by  the  in-field  QC  software. 

During  the  field  demonstration  component  of  this  project,  we  supplied  the  Parsons  field  team 
with  the  in-field  QC  software  during  their  cued  MetalMapper  survey  at  the  Former  Waikoloa 
Maneuver  Area  (WMA)  on  the  island  of  Hawaii.  During  this  survey,  the  field  team  encountered 
1032  unique  anomaly  locations  with  the  MetalMapper.  Out  of  these  1032  encounters,  231 
resulted  in  recollects  based  on  the  estimated  target  location  feedback  provided  by  the  in-field  QC 
software. 

We  performed  a  retrospective  analysis  of  these  MetalMapper  data  to  determine: 

•  if  there  were  any  missed  recollect  opportunities,  i.e.,  cases  where  a  recollect  was  not 
performed,  but  should  have  been  performed; 

•  the  effectiveness  of  the  in-field  QC  process  by  quantifying  any  improvements  in  target 
features  obtained  by  recollecting  the  data; 

•  the  efficiency  of  the  in-field  QC  process  by  identifying  the  number  of  cases  where  the 
recollect  was  unnecessary,  i.e.,  it  did  not  produce  better  characterization  of  the  target. 

We  used  these  recollect  statistics  to  develop  estimates  of  production  rates  for  surveys  conducted 
using  the  in-field  QC  approach  and  for  surveys  where  no  in-field  recollect  decision  is  made.  A 
summary  of  the  statistics  and  estimated  production  rates  are  as  follows: 

•  out  of  1032  anomalies  investigated  of  which  231  resulted  in  a  recollect,  there  was  1 
potential  missed  recollect  opportunity; 

•  out  of  231  recollects,  153  recollects  appeared  to  be  a  result  of  magnetic  geology  creating 
false  source  locations; 

•  out  of  the  remaining  78  recollects  that  were  due  to  legitimate  sources  (i.e.,  a  metal 
object),  46  resulted  in  improvements  in  target  characterization; 

•  of  the  remaining  32  recollects  that  did  not  significantly  improve  target  characterization, 
we  found  11  cases  where  the  unnecessary  recollect  may  have  been  avoided  with  the 
application  of  additional  quality  metrics  (i.e.,  in  addition  to  the  estimated  target  location 
metric); 

•  Estimated  production  rates  for  surveying  with  and  without  the  in-field  QC  process  were 
23  anomalies/hr  (with  in-field  QC)  and  26  anomalies/hr  (without  in-field  QC). 
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The  magnetic  geology  at  the  site  presented  the  most  significant  challenge  to  the  technology  and 
contributed  to  the  lower  than  expected  production  rate  for  the  in-field  QC  approach.  The  46 
cases  that  resulted  in  quantifiable  improvements  in  target  features  are  an  example  of  the  potential 
benefits  of  applying  in-field  QC  to  cued  EMI  surveys.  Possible  ways  to  improve  the  efficiency 
of  the  technology  at  challenging  sites,  such  as  the  WMA,  could  be  to  improve  background 
selection  and  removal  during  in-field  QC  of  the  data  or  to  implement  multi-source  solvers  in  the 
in-field  inversion  to  account  for  magnetic  soil  effects  and  high  target  densities. 
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1.  INTRODUCTION 

This  project  was  undertaken  to  evaluate  and  implement  quality  control  processes  for  cued 
Electromagnetic  Induction  (EMI)  data  collection  associated  with  munitions  classification 
surveys.  The  primary  objective  of  this  effort  was  to  establish  an  optimal  set  of  metrics  that  will 
inform  field  teams  of  the  classification  quality  of  the  data  collected  during  a  cued  EMI  survey. 
Incorporating  these  metrics  in  a  software  interface  will  enable  field  operators  to  determine 
immediately  upon  acquisition  if  cued  data  are  of  high  enough  quality  to  provide  useful 
classification  features.  This  ability  to  make  an  informed  in-field  decision  regarding  cued  EMI 
data  quality  has  the  potential  to  greatly  increase  the  efficiency  of  these  surveys.  Currently,  most 
of  the  quality  analysis  for  cued  EMI  surveys  is  performed  off-site  by  trained  analysts.  While 
effective  for  identifying  instances  of  poor  data  quality,  this  method  can  be  costly  as  it  often 
requires  field  teams  to  redeploy  to  certain  areas  to  reacquire  data.  Enabling  an  in-field  quality 
decision  will  produce  immediate  corrective  actions  that  will  obviate  the  need  for  subsequent 
redeployment  and  reacquisition  by  the  field  team. 

During  the  first  phase  of  this  project,  we  evaluated  an  in-field  Quality  Control  (QC)  software 
module  that  incorporates  a  real-time  inversion  capability  to  provide  field  operators  with  a  set  of 
quality  metrics  that  allows  them  to  make  high  confidence  quality  decisions  during  cued  surveys. 
If  a  low  quality  data  collection  event  occurs,  the  software  module  also  facilitates  immediate 
corrective  action  by  presenting  the  operators  with  a  set  of  rectified  sensor  positioning 
coordinates.  The  QC  module  is  currently  designed  to  support  MetalMapper  operations;  however, 
it  can  be  modified  to  work  with  any  advanced  EMI  sensor.  As  part  of  this  initial  phase  we 
completed  a  series  of  proveout  tests  to  verify  the  performance  and  functionality  of  the  QC 
module  applied  to  cued  MetalMapper  data  collection.  Results  from  these  tests  demonstrated  that 
the  software  module  performance  met  the  objectives  required  for  transition  to  field  operations. 
These  objectives  included  criteria  for  the  reliability  of  inversion  parameters  as  well  as 
requirements  for  the  sensor  navigation  and  positioning  accuracy. 

The  second  phase  of  this  project  provided  a  field  demonstration  of  the  QC  software  module  and 
associated  quality  control  practices  at  the  Former  Waikoloa  Maneuver  Area  (WMA).  The  field 
component  of  this  demonstration  involved  the  QC  module  and  in-field  quality  control  practices 
applied  to  a  MetalMapper  survey  conducted  by  a  field  team  at  a  live  munitions  site.  This  field 
demonstration  helped  to  elucidate  some  of  the  practical  and  operational  requirements  for 
conducting  efficient  data  quality  control  by  the  field  team.  It  also  provided  a  means  for 
acquiring  a  substantial  data  set  that  can  be  used  to  quantify  the  overall  effectiveness  of  the  in¬ 
field  quality  control  process.  During  our  retrospective  analysis  of  the  WMA  data  set,  we 
identified  several  instances  where  the  in-field  QC  led  to  improvements  in  the  data  quality,  as 
well  as  cases  where  it  led  to  unnecessary  recollection  of  cued  data.  These  cases  can  be  taken 
together  to  provide  some  insight  into  the  benefits  and  limitations  of  the  technology,  as  well  as  to 
reveal  possible  approaches  for  improving  the  effectiveness  of  the  in-field  QC  approach.  This 
document  provides  an  overview  of  the  technology  and  in-field  QC  approach,  a  summary  of  the 
performance  results  from  our  retrospective  analysis  of  the  WMA  data,  and  suggestions  for 
potential  enhancements  to  the  in-field  QC  approach. 
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1.1  BACKGROUND 

The  prevalence  of  innocuous  clutter  (e.g.,  scrap,  fragmentation,  etc.)  at  Munitions  Response 
(MR)  sites  presents  a  challenge  to  remediation  efforts  that  often  devote  substantial  resources  to 
the  excavation  and  identification  of  these  non-hazardous  objects.  Over  the  last  5-10  years,  the 
development  of  advanced  EMI  sensor  arrays  that  enable  multi-axis  or  multi-angle  illumination  of 
cued  anomalies  has  enabled  the  implementation  of  methodologies  that  effectively  discriminate 
clutter  from  Unexploded  Ordnance  (UXO)  or  other  Munitions  and  Explosives  of  Concern 
(MEC).  These  classification  methodologies  have  the  potential  to  significantly  improve  the 
efficiency  of  production  cleanup  efforts  by  reducing  the  time  and  costs  associated  with  removal 
of  benign  objects;  however,  collection  of  high  quality  data  with  advanced  sensors  is  critical  to 
ensuring  the  effectiveness  of  these  classification  algorithms. 

Currently,  standard  protocol  for  collecting  advanced  EMI  sensor  data  includes  static  data 
acquisition  over  anomalies  identified  (cued)  by  prior  dynamic  or  Digital  Geophysical  Mapping 
(DGM)  surveys.  These  data  are  typically  analyzed  by  off-site  geophysicists  upon  completion  of 
the  cued  survey.  Often,  this  analysis  results  in  a  number  of  anomalies  that  are  found  to  be 
insufficiently  characterized  by  the  initial  cued  data  collection  as  a  result  of  sub-optimal 
placement  of  the  EMI  sensor  during  the  survey.  Subsequently,  reacquisition  of  data  is  necessary 
and  additional  surveying  is  required.  Methods  that  significantly  reduce  reacquisitions  and 
improve  the  overall  quality  of  the  data  before  completion  of  the  initial  cued  survey  will 
significantly  improve  the  efficiency  of  production  operations. 

Because  cued  EMI  sensors  rely  on  multi-axis  illumination  to  provide  enough  information  to 
extract  useful  classification  features  from  the  data,  the  placement  of  the  sensor  relative  to  the 
object  under  interrogation  is  critical.  In  many  instances  when  the  cued  data  yield  poor 
classification  features,  the  reason  is  often  an  incorrectly  positioned  sensor.  Positioning  errors, 
misguided  target  picking,  or  multiple  objects  in  the  field  of  view  are  common  causes  of  incorrect 
cued  sensor  placement.  Most  often,  these  errors  are  not  discovered  until  the  off-site  geophysicist 
identifies  them  during  analysis  of  the  data.  Once  discovered,  the  analyst  will  determine  a 
rectified  set  of  coordinates  to  better  represent  the  true  location  of  the  target.  The  field  team  will 
then  reacquire  the  data  by  positioning  the  sensor  over  the  new  coordinates. 

Providing  the  field  team  with  the  capability  to  assess  data  quality  immediately  following  the 
initial  data  collection  could  significantly  improve  the  cued  survey  process.  By  removing  the  off¬ 
site  analyst  from  the  initial  quality  decision  making,  the  field  team  can  take  immediate  action  to 
reposition  and  recollect  certain  targets  if  deemed  necessary.  Immediate  recollection  based  on  in¬ 
field  quality  control  can  be  much  more  efficient  than  subsequent  reacquisition  based  on  off-site 
decision  making.  Offsite  analysis  may  still  be  important  for  identifying  complicated  target 
scenarios  (e.g.,  multi-object,  magnetic  geology,  etc.);  however,  many  of  the  reacquires  due  to 
common  errors  such  as  inaccurate  target  picking  could  be  replaced  by  in-field  decision  based 
recollects. 

By  incorporating  the  QC  software  module  into  the  existing  cued  EMI  survey  process,  we 
endeavored  to  establish  a  set  of  procedures  that  will  bridge  the  current  capability  gaps  associated 
with  in-field  quality  control  of  munitions  classification  data.  The  QC  module  provides  the  field 
team  with  data  inversion  parameters  such  as  target  polarizabilities  and  estimated  target  location 
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while  the  EMI  sensor  is  still  in  the  vicinity  of  cued  anomaly  locations.  These  parameters  may 
then  be  used  to  identify  anomalies  that  are  poorly  characterized  due  to  sub-optimal  sensor 
positioning.  Accordingly,  the  field  team  can  take  appropriate  and  immediate  corrective  action 
before  leaving  the  survey  area.  This  process  can  significantly  improve  the  overall  efficiency  of 
production  classification  surveys  by  reducing  the  need  for  later  redeployment  and  reacquisition. 

1.2  OBJECTIVE  OF  THE  DEMONSTRATION 

During  the  initial  phase  of  project  MR-201264  we  verified  the  basic  performance  capabilities  of 
the  QC  module.  This  verification  process  included  analysis  of  controlled  test  pit  data  to  quantify 
the  accuracy,  speed,  and  reliability  of  the  QC  module  output.  We  also  conducted  a  preliminary 
evaluation  of  the  process  flows  associated  with  the  in-field  QC  protocol  by  performing  cued 
surveys  over  surrogate  test  items.  This  evaluation  yielded  a  qualitative  assessment  of  the 
operational  aspects  of  the  QC  process  such  as  ease  of  use  and  level  of  operator  interpretation 
required.  The  field  demonstration  phase  of  this  project  tested  the  performance,  reliability,  and 
operational  feasibility  of  this  QC  process  under  more  demanding  field  conditions. 

The  primary  objective  of  the  demonstration  phase  of  this  project  was  to  gain  further  insight  into 
the  field  practices  that  lead  to  the  most  effective  and  efficient  cued  EMI  surveys.  For  this 
demonstration  phase,  we  worked  with  a  field  team  to  apply  the  QC  module  during  a  cued  survey 
of  the  Former  WMA  to  identify  anomalies  that  may  have  been  insufficiently  characterized  by  the 
initial  data  collection.  After  the  survey  and  final  ground  truth  stages  were  completed,  we 
performed  a  retrospective  analysis  of  the  WMA  data  set  to  identify  cases  where  recollects  based 
on  the  in-field  decision  led  to  an  improvement  in  data  quality  as  well  as  cases  where  the  recollect 
was  unnecessary  (i.e.,  it  did  not  provide  any  improvements  in  classification  features). 

Other  aspects  of  the  demonstration  that  helped  us  gauge  the  overall  utility  of  the  QC  module  in 
supporting  field  operations  included  anecdotal  evidence  (based  on  the  feedback  supplied  by 
several  MetalMapper  operators),  which  suggests  that  the  QC  module  navigation  interface  offers 
a  very  intuitive  approach  to  sensor  positioning.  In  fact,  user  feedback  indicates  that  the  time 
spent  performing  an  initial  data  collection  as  well  as  an  immediate  sensor  reposition  and 
recollection  using  the  QC  module  is  often  less  than  the  time  required  to  perform  only  an  initial 
collection  without  using  the  feedback  provide  by  the  QC  module.  This  result  has  implications 
for  survey  production  rates.  It  is  possible  that  by  reducing  the  level  of  operator  interpretation 
required  to  position  the  sensor,  the  corresponding  production  rate  will  increase  even  when  cases 
of  recollection  are  accounted  for  (i.e.,  production  rates  are  based  on  unique  acquisitions  per  day 
as  opposed  to  total  acquisitions  per  day).  While  a  number  of  factors  influence  production  rate 
(e.g.,  anomaly  density,  terrain  adversity,  operator  skill,  etc.),  a  field  demonstration  of  this 
technology  nevertheless  provides  some  indication  of  how  the  QC  module  might  improve  the 
efficiency  of  daily  operations.  By  comparing  the  production  rate  associated  with  the  in-field  QC 
process  to  the  production  rates  achieved  during  past  demonstrations,  we  can  get  a  sense  of  the 
overall  utility  of  this  approach  regardless  of  the  other  influencing  factors. 

By  testing  the  in-field  QC  process  in  a  live  site  venue,  this  demonstration  allowed  us  to  quantify 
the  benefits  of  the  approach  as  well  as  identify  practical  strategies  for  improving  the  utility  and 
effectiveness  of  the  approach.  The  Former  WMA  site  presented  a  unique  set  of  challenges 
associated  with  magnetic  soils;  however,  the  suggested  improvements  to  the  in-field  QC  process 
can  be  applied  generally  to  sites  where  background  soil  response  may  be  variable. 
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1.3  REGULATORY  DRIVERS 

Military  Munitions  Response  Program  (MMRP)  regulations  require  well  defined  Standard 
Operating  Procedures  (SOPs)  for  DGM  surveys  conducted  throughout  MR  sites.  These  SOPs 
include  guidelines  for  conducting  daily  verification  of  instrument  functionality  for  both 
geophysical  and  positioning  equipment  used  during  the  survey.  Procedures  are  also  defined  for 
the  processing  and  analysis  of  geophysical  data  to  ensure  that  the  survey  produces  high  quality 
data  that  will  indicate  the  location  of  any  potential  contaminants.  While  the  SOPs  associated 
with  DGM  operations  have  been  refined  through  years  of  practice  to  produce  effective  survey 
results,  the  introduction  of  cued  classification  surveys  to  MMRP  projects  will  require  significant 
modifications  to  these  procedures. 

Classification  surveys  that  rely  on  advanced  EMI  sensors  require  particularly  strict  guidelines  for 
sensor  positioning.  This  requirement  is  due  to  the  fact  that  advanced  sensors  capture  multi-axis 
data  that  are  most  useful  if  they  contain  information  about  the  target’s  principal  components. 
Placement  of  the  sensor  relative  to  the  anomaly  is  critical  for  ensuring  complete  characterization 
of  the  target  physical  properties.  Additionally,  the  data  analysis  associated  with  classification 
sensors  is  arguably  more  complex  than  DGM  analysis.  While  DGM  data  processing  includes 
such  steps  as  filtering  and  gridding,  there  is  no  incorporation  of  physical  models  in  the  process. 
Classification  analysis  requires  the  inversion  of  data  through  comparison  to  physics-based 
models  that  represent  the  principal  object  properties.  Accordingly,  the  data  quality  requirements 
for  classification  data  are  much  different  than  the  typical  threshold  quality  objectives  used  for 
DGM  data. 

The  in-field  QC  module  provides  the  field  team  with  an  intuitive  means  for  assessing  not  just 
basic  instrument  functionality,  but  also  classification  quality  of  the  data.  Without  this  immediate 
feedback,  the  field  team  must  rely  on  off-site  analysis  before  knowing  if  data  quality  objectives 
are  achieved.  Thus,  providing  the  field  team  with  relevant  quality  metrics  is  extremely  important 
for  ensuring  that  classification  surveys  are  conducted  in  an  efficient  manner  and  meet  the  data 
quality  objectives  that  are  specific  to  these  surveys. 

2.  TECHNOLOGY 

The  basis  for  the  QC  module  technology  is  a  C++  inversion  and  classification  Application 
Programming  Interface  (API)  that  provides  access  to  physical  model  (i.e.,  dipole)  inversion 
algorithms,  which  can  be  used  to  rapidly  estimate  a  variety  of  model  parameters  associated  with 
each  anomaly.  The  QC  module  allows  the  field  team  to  obtain  information  about  target  physical 
properties  immediately  after  acquiring  a  cued  EMI  data  file.  This  information  can  then  be  used 
to  assess  the  quality  of  the  data  based  on  the  reliability  of  these  target  features. 

The  following  subsection  presents  a  concise  overview  of  the  QC  module  technology  including 
the  procedures  associated  with  its  use.  A  more  detailed  description  of  the  technology  is 
presented  in  the  MR-201264  Interim  Proveout  Report  submitted  to  the  ESTCP  Program  Office 
(included  in  Appendix  A). 

2.1  TECHNOLOGY  DESCRIPTION 

The  QC  module  functions  as  a  parallel  process  to  the  cued  EMI  sensor  data  acquisition  software. 
As  such,  the  QC  module  does  not  provide  any  data  acquisition  functionality;  it  is  used  solely  for 
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immediate  in-field  analysis  of  the  cued  sensor  data.  The  QC  module  uses  a  file  watch  protocol, 
which  means  that  it  loads  cued  sensor  data  files  immediately  after  they  are  created  in  a  data 
acquisition  file  directory.  In  this  sense,  the  QC  software  is  truly  modular  in  that  it  can  be  used  in 
parallel  with  any  data  acquisition  software,  provided  the  associated  data  file  formats  are 
accepted. 

The  existing  sensor  data  acquisition  software  (e.g.,  MetalMapper  EM3DAcquire)  provides  the 
operator  with  the  appropriate  interfaces  to  set  the  acquisition  parameters  and  trigger  an 
acquisition  event;  however,  all  other  processes  associated  with  the  survey  including  sensor 
navigation  and  data  quality  analysis  are  facilitated  by  the  QC  module.  Because  the  ability  to 
reposition  the  sensor  head  based  on  data  inversion  results  is  a  key  component  of  in-field  QC,  it 
was  a  logical  step  to  include  this  capability  as  part  of  a  QC  navigation  interface  that  functioned 
separately  from  the  data  acquisition  software.  Additionally,  any  cued  sensor  will  require  sensor 
head  positioning  information,  typically  in  the  form  of  GPS  and  Inertial  Measurement  Unit  (IMU) 
data  streams.  It  was  a  relatively  straightforward  process  to  split  these  incoming  positioning  data 
streams  between  the  sensor  data  acquisition  software  and  the  QC  module. 

When  the  QC  module  executable  performs  a  file  watch  on  the  cued  data  directory,  any 
acquisition  event  triggered  by  the  sensor  acquisition  software  interface  will  create  a  new  file  in 
this  directory.  The  QC  module  will  immediately  load  this  file  and  invert  the  data  contained 
within  it.  The  output  of  these  inversions  is  subsequently  transferred  to  the  operator  navigation 
and  QC  interfaces  that  display  the  various  recovered  dipole-model  parameters  (e.g., 
polarizabilities,  location,  depth,  etc.)  as  well  as  the  raw  data  channel  output. 

In  practice,  operating  the  sensor  data  acquisition  software  and  the  QC  module  in  parallel  is 
straightforward  and  does  not  require  any  shuffling  of  interfaces  to  acquire  data.  Our  initial 
concern  in  having  separate  data  acquisition  and  data  QC  processes  was  that  it  might  be  onerous 
for  the  operator  to  switch  back  and  forth  between  the  data  acquisition  interface  for  triggering  an 
acquisition  event  and  the  QC  interface  for  viewing  results;  however,  in  reality  this  process 
requires  very  few  steps.  We  assessed  these  operational  flows  by  integrating  the  software  with  a 
production  MetalMapper  system.  Once  the  initial  data  acquisition  parameters  are  set  for  the 
MetalMapper  using  the  EM3D  acquisition  software,  the  QC  module  can  be  used  to  provide  the 
primary  navigation  and  quality  analysis  interfaces.  Only  a  small  trigger  button  from  the  EM3D 
interface  must  remain  accessible  for  commencing  a  data  acquisition  event.  Figure  1  shows  a 
screenshot  capturing  a  typical  arrangement  of  these  interfaces.  It  is  possible  to  make  both  the 
data  acquisition  and  QC  functions  available  within  the  same  display. 
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Figure  1.  Typical  operator  display  setup  showing  QC  and  acquisition  interfaces.  The  QC  Module  provides 
the  main  navigation  and  analysis  interfaces,  while  EM3D  provides  the  acquisition  capability  with  a  simple 
acquire  button. 

The  QC  module  navigation  interface  facilitates  sensor  positioning  and  data  recollects  by 
displaying  target  coordinates  relative  to  the  cued  EMI  sensor  head  location.  Target  picks 
identified  from  analysis  of  the  DGM  data  can  be  loaded  into  the  navigation  interface  to  show  the 
location  of  each  anomaly  that  will  be  interrogated  with  the  cued  EMI  sensor.  Once  the  operator 
selects  an  anomaly  for  interrogation,  the  navigation  interface  will  display  both  the  anomaly 
location  and  the  sensor  head  location  (Figure  2).  As  the  operator  maneuvers  the  sensor  head 
closer  to  the  anomaly,  the  navigation  display  auto-zooms  to  enable  accurate  positioning  of  the 
sensor  over  the  anomaly.  The  display  provides  a  sensor  frame-of-reference  view  such  that  the 
sensor  heading  is  always  directed  toward  the  top  of  the  operator  display  screen.  Once  the  sensor 
head  is  positioned  over  the  anomaly  coordinates,  the  operator  acquires  a  data  file.  Immediately, 
the  data  are  inverted  and  the  estimated  coordinates  of  the  target  are  displayed.  If  necessary  the 
operator  can  then  reposition  the  sensor  using  these  new  coordinates. 
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Figure  2.  LEFT:  The  navigation  interface  displays  the  anomaly  coordinates  and  the  sensor  location.  RIGHT: 
As  the  operator  moves  the  sensor  closer  to  an  anomaly,  the  display  performs  an  auto-zoom  to  facilitate 
accurate  positioning  of  the  sensor  head  over  the  anomaly.  After  the  data  are  acquired,  the  QC  module 
updates  the  navigation  display  with  the  estimated  target  location  (pink  dot). 


In  addition  to  providing  the  operator  with  the  estimated  target  location,  the  QC  module  also 
provides  information  about  model  parameters  that  correspond  to  target  features.  Classification 
features,  such  as  target  polarizabilities,  are  compared  to  those  catalogued  in  a  target  library.  This 
library  corresponds  to  possible  targets  of  interest  (TOIs)  that  may  be  located  at  the  site.  The  TOI 
library  is  built  from  data  sets  collected  in  controlled  calibration  areas  within  the  site  (Figure  3). 
Cued  EMI  data  are  collected  in  a  calibration  area  (i.e.,  a  test  pit)  over  TOIs  that  are  indigenous  to 
the  site.  Features  from  these  TOIs  are  then  added  to  the  library.  During  the  subsequent  cued 
survey,  the  QC  interface  will  display  the  polarizabilities  corresponding  to  each  anomaly  item  and 
compare  them  to  the  library  features.  The  QC  interface  also  indicates  the  best  match  TOI.  The 
library  can  be  expanded  with  additional  data  sets  collected  throughout  the  survey  if  additional 
items  of  interest  are  found. 


Figure  3.  LEFT:  Target  libraries  are  built  from  data  collected  in  controlled  calibration  areas  within  the  site. 
This  image  shows  a  37mm  projectile  placed  in  a  test  pit  for  library  data  collection.  RIGHT:  Library  features 
are  subsequently  loaded  into  the  QC  module.  The  QC  interface  displays  polarizabilities  recovered  from  cued 
EMI  data  inversion  and  compares  them  to  the  best-match  library  features. 
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We  performed  extensive  tests  to  quantify  the  performance  of  the  QC  module  in  terms  of  speed, 
accuracy,  and  reliability.  One  of  the  key  requirements  for  production  use  is  that  the  time 
required  to  invert  the  data  and  present  results  is  negligible  in  comparison  to  the  total  time  to 
acquire  a  data  file  (approximately  10-15  seconds  for  a  cued  MetalMapper  file).  We  conducted 
several  operational  tests  that  demonstrated  the  inversion  of  data  in  each  file  typically  requires  0.2 
-  0.4  seconds  to  complete  and  produce  results.  We  clocked  the  inversion  time  for  many  sample 
data  files  and  found  these  results  were  highly  repeatable.  When  incorporated  as  part  of  the  data 
acquisition  cycle,  this  inversion  step  has  a  minimal  impact  on  the  overall  process  flow. 

We  also  verified  the  accuracy  and  reliability  of  the  navigation  functions.  Through  careful  and 
controlled  measurements,  we  found  that  the  navigation  interface  provided  enough  positioning 
accuracy  to  guide  the  sensor  head  within  2-3  cm  of  the  desired  target  pick  location. 
Additionally,  we  found  that  once  a  data  file  was  acquired,  the  inversion  provided  an  estimated 
target  location  within  5  cm  of  the  true  location  (verified  by  careful  test  grid  measurements).  The 
reliability  of  the  estimated  target  location  parameter  was  demonstrated  for  a  range  of  lateral 
offsets  between  the  target  item  and  the  center  of  the  MetalMapper  sensor  head.  We  found  that 
the  accuracy  of  the  target  location  parameter  was  maintained  even  for  target  offsets  extending  to 
70  cm  from  the  sensor  center.  Figure  4  shows  an  example  of  the  results  from  these  performance 
tests. 


Figure  4.  LEFT:  QC  module  performance  tests  were  conducted  by  acquiring  MetalMapper  measurements 
over  target  items  placed  on  a  test  grid.  CENTER:  Sensor  positioning  guided  by  the  navigation  interface  was 
demonstrated  to  be  within  2-3  cm  of  target  pick  location  (red  dot).  RIGHT:  Estimated  target  location 
determined  by  data  inversion  was  within  5  cm  of  the  true  target  location  for  target  offsets  extending  to  70  cm 
from  the  center  of  the  MetalMapper  sensor  head. 

After  verifying  the  three  operational  requirements,  we  tested  the  QC  module  functionality  during 
a  mock  cued  survey  using  the  MetalMapper  system.  We  placed  targets  near  several  cued 
locations  that  we  incorporated  in  a  target  pick  file.  Once  these  pick  locations  were  loaded  into 
the  QC  module,  we  performed  a  cued  survey  at  each  location.  For  each  object,  the  initial  sensor 
positioning  and  any  subsequent  repositioning  were  based  entirely  on  the  feedback  from  the  QC 
module.  Thus,  the  operator’s  quality  decision  was  not  based  on  any  knowledge  of  the  location  or 
identity  of  the  object  under  interrogation.  An  example  of  the  resulting  in-field  QC  analysis 
corresponding  to  one  of  these  interrogations  is  shown  in  Figure  5. 
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Figure  5.  TOP  LEFT:  The  red  dot  shows  the  location  of  the  target  pick.  The  pink  dot  shows  the  estimated 
target  location.  TOP  CENTER:  The  sensor  is  repositioned  over  the  estimated  location,  which  is  now 
represented  by  the  red  dot.  TOP  RIGHT:  If  the  operator  determines  the  quality  of  the  recollected  data  is 
acceptable,  the  dot  turns  green  indicating  that  the  data  corresponding  to  this  anomaly  have  met  the  quality 
requirements.  BOTTOM  LEFT:  QC  interface  for  the  recollected  data.  BOTTOM  RIGHT:  The  actual 
target  (37  mm  projectile)  and  target  pick  location  (orange  paint). 


2.2  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGY 

The  QC  Module  provides  several  advantages  for  field  teams  conducting  cued  EMI  surveys. 
Currently,  there  are  no  commercially  available  software  packages  that  directly  link  with 
advanced  sensor  output  to  provide  data  inversion  results  in  real-time  (i.e.,  immediately  after  the 
acquisition  of  a  data  file).  The  capability  for  sensor  operators  to  obtain  inversion  parameters 
such  as  estimated  target  location  and  target  polarizabilities  while  the  sensor  is  still  in  proximity 
to  the  target  pick  location  has  significant  implications  for  production  rates.  By  providing  the 
operator  with  the  necessary  information  to  assess  data  quality,  the  QC  module  facilitates 
immediate  corrective  action  based  on  in-field  decisions. 


An  alternative  approach  to  in-field  QC  is  to  rely  on  direct  sensor  output  (in  lieu  of  inversion 
output)  to  determine  data  quality.  This  approach  is  currently  a  standard  practice  for 
MetalMapper  surveys.  The  MetalMapper  data  acquisition  software  provides  the  operator  with 
visual  interpretation  of  the  vector  magnetic  field  measured  in  each  sensor  receiver.  This 
information  can  be  used  to  help  guide  the  sensor  to  the  target  location.  While  this  method  can  be 
effective,  it  does  not  provide  the  level  of  information  that  is  contained  in  the  inversion  output. 
The  raw  sensor  output  makes  it  difficult  to  separate  the  target  pick  anomaly  response  from  the 
influence  of  nearby  objects.  This  influence  can  sometimes  result  in  the  misplacement  of  the 
sensor  over  smaller  clutter  objects.  Also,  as  previously  mentioned  our  conclusions  based  on 
feedback  from  several  operators  indicate  that  the  interpretation  of  inversion  results  is  more 
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intuitive  and  can  produce  a  more  efficient  survey  than  the  interpretation  based  on  the  raw  sensor 
output. 

The  primary  limitation  of  the  technology  is  the  current  lack  of  a  real-time  multi-object  inversion 
capability.  During  our  preliminary  tests,  we  found  that  for  QC  purposes  the  single-object 
inversion  works  fairly  well  for  many  cases  involving  multiple  objects  within  the  sensor’s  field  of 
view;  however,  the  effectiveness  of  the  single-object  approach  depends  largely  on  the  size, 
proximity,  and  number  of  influencing  objects.  One  of  the  objectives  of  the  demonstration  was  to 
identify  the  limitations  of  our  approach  in  regards  to  multi-object  cases.  Certain  data  or  model 
parameters  may  be  good  indicators  of  whether  multi-source  solvers  are  necessary.  Thus,  it  might 
be  possible  to  flag  data  files  that  are  likely  to  require  multi-object  analysis  before  the  analyst 
receives  them.  Currently,  our  approach  still  relies  on  the  off-site  analyst  to  make  decisions 
regarding  more  complex  multi-object  cases;  however,  we  anticipate  a  future  variant  of  the  QC 
module  could  accommodate  multi-source  scenarios. 

Another  potential  limitation  that  became  apparent  during  demonstration  of  the  technology  at  the 
Former  Waikoloa  Maneuver  Area  is  the  interference  of  magnetic  soils  with  the  real-time 
inversion  output.  Retrospective  analysis  of  the  Waikoloa  data  indicated  that  in  many  instances, 
the  in-field  inversion  produced  source  locations  that  were  due  to  geology.  Mitigating  the  effects 
of  magnetic  soils  poses  a  significant  challenge  not  just  for  in-field  analysis,  but  for  off-site 
analysts  as  well.  It  may  be  difficult  to  remove  these  effects  entirely  during  the  in-field  analysis; 
however,  possible  solutions  could  include  more  frequent  background  updates  to  the  in-field 
software  or  the  implementation  of  a  decay  fit  to  indicate  to  the  field  operator  if  the  response  is 
likely  due  to  ground  conditions. 

3.  PERFORMANCE  OBJECTIVES 

The  basis  for  the  field  portion  of  the  demonstration  consisted  of  a  cued  (static)  MetalMapper 
survey  of  anomalies  identified  from  a  prior  EM-61  DGM  survey.  The  MetalMapper  field  team 
conducted  a  survey  of  approximately  1000  anomalies  while  using  the  in-field  QC  module.  The 
team  conducted  recollects  for  anomaly  locations  where  the  in-field  software  indicated  the 
presence  of  a  source  that  was  more  than  30cm  from  the  center  of  the  MetalMapper.  These 
procedures  led  to  231  recollects  over  the  course  of  the  survey.  These  recollects  formed  the  basis 
for  our  retrospective  analysis  of  the  QC  performance.  With  the  incorporation  of  the  QC  module 
into  the  survey  protocol,  there  were  five  possible  scenarios  that  can  be  used  to  establish  the 
capabilities  and  limitations  of  the  technology.  These  scenarios  included: 

1.  Cases  where  no  recollect  was  taken  and  no  recollect  was  necessary.  These  cases 
accounted  for  about  80%  of  the  total  survey.  While  these  instances  did  not  contribute  to 
any  quantitative  assessment  of  the  technology  performance,  they  can  be  used  as  a 
baseline  to  assess  the  qualitative  aspects  associated  with  ease  of  use  and  production 
efficiency. 

2.  Cases  where  no  recollect  was  taken,  but  was  necessary.  Such  instances  are  representative 
of  the  technology  limitations  (i.e.,  cases  where  the  in-field  QC  process  failed  to  correctly 
identify  a  recollect  opportunity).  Out  of  1032  anomalies,  we  identified  one  case  where 
repositioning  and  recollecting  may  have  been  useful.  A  possible  missed  recollect 
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opportunity  occurred  for  seed  item  WK-1047,  where  the  sole  MetalMapper  acquisition 
was  collected  at  an  offset  of  43cm  from  the  seed  location.  Seed  WK-1047  was  one  of  the 
two  seeds  missed  in  the  classification  stage  by  the  project  analysts.  While  it  is  unclear  if 
repositioning  and  recollecting  would  have  improved  classification  features,  a  43cm  offset 
would  normally  produce  a  recollect  decision.  Most  likely  this  recollect  opportunity  was 
missed  because  of  the  very  strong  soil  response  in  this  area  (post-survey  analysis  of  this 
anomaly  produced  ground-like  features),  which  may  have  produced  inaccurate  in-field 
source  estimates. 

3.  Cases  where  a  recollect  was  taken,  was  necessary,  and  improved  the  classification 
features.  This  scenario  enables  a  quantitative  assessment  of  the  technology  performance 
by  providing  clear  examples  of  the  advantages  afforded  by  the  in-field  QC  process.  Our 
analysis  indicated  that  there  were  46  cases  where  the  recollect  improved  classification 
features  associated  with  the  anomaly  (in  some  cases  this  improvement  was  significant 
enough  that  without  the  recollect,  correct  classification  may  have  been  difficult). 

4.  Cases  where  the  in-field  model  did  not  provide  accurate  characterization  of  the  survey 
space.  This  scenario  could  include,  for  example,  multi-object  cases  where  the  QC 
module  correctly  identifies  a  problematic  target,  but  does  not  provide  the  in-field 
capability  to  characterize  the  situation  correctly;  however,  for  the  WMA  survey,  the 
greatest  challenge  to  the  in-field  QC  models  was  presented  by  the  magnetic  geology.  Out 
of  the  231  recollects  conducted  in  the  WMA  survey,  153  of  these  appeared  to  be  caused 
by  the  in-field  models  indicating  a  ground  source  (114  of  these  occurred  in  areas  where 
no  target  was  present,  39  occurred  in  areas  where  small  debris  was  found). 

5.  Cases  where  a  recollect  was  taken,  but  was  unnecessary.  In  other  words,  this  scenario 
comprises  cases  where  the  initial  acquisition  provided  sufficient  classification  features. 
During  retrospective  analysis,  we  found  32  cases  where  the  initial  acquisition  produced 
accurate  features  for  classification  and  the  recollect  did  not  offer  any  significant 
improvements.  These  were  cases  where  the  30cm  recollect  rule  was  likely  overly 
conservative. 

Each  of  the  aforementioned  scenarios  is  associated  with  specific  performance  objectives.  During 
our  post-survey  analysis,  we  applied  a  set  of  metrics  that  correspond  to  each  of  these  objectives. 
Table  1  summarizes  the  performance  objectives  and  corresponding  metrics  used  during 
retrospective  analysis. 


Table  l.  Performance  Objectives 


Performance 

Objective 

Metric 

Data  Required 

Success  Criteria 

Quantitative  Performance  Objectives 

Identification  of 
all  recollect 
opportunities 

Percent  of  reacquires 
out  of  total  recollect 
opportunities 

•  Off-site  QC  analysis 
of  all  MM  initial 
acquisition  files 

PreaccO.  1 
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Effective 
corrective  action 

Percent  recollects  out  of 
total  recollects  resulting 
in  improved  target 
features 

•  Off-site  QC  analysis 
of  MM  recollects 
and  corresponding 
initial  acq.  files 

Prec(I)>0.9 

Effective  in-field 
characterization 

Number  of  ineffective 
recollects  due  to 
inadequate  in-field 
models 

•  MM  recollect  files 
that  did  not  provide 
improved  features 

Insufficient  model 
characterization  <5% 
of  total  recollect  cases 

Effective  quality 
metrics 

Number  of  ineffective 
recollects  eliminated 

•  MM  recollect  files 
that  did  not  provide 
improved  features 

Any  reduction  in 
ineffective  recollects 

Production  rate 

Number  of  unique 
anomalies  surveyed  per 
day  (in  terms  of  hourly 
quotas) 

•  Field  logs,  data  file 
time  stamps 

Site  dependent 

Qualitative  Performance  Objectives 

Ease  of  use 

•  Operator  feedback 
regarding 
intuitiveness  of 
display,  QC  results, 
and  process  flows 

3.1  OBJECTIVE:  IDENTIFICATION  OF  ALL  RECOLLECT  OPPORTUNITIES 

Our  goal  was  to  use  the  QC  module  to  identify  any  acquisition  that  could  be  improved  by 
repositioning  the  sensor.  This  in-field  QC  decision  was  based  on  the  estimated  target  location 
parameter.  If  the  estimated  target  location  was  greater  than  30  cm  from  the  center  of  the  sensor 
(R>0.30m),  the  field  team  would  reposition  the  sensor  and  conduct  another  cued  sounding.  This 
30cm  radius  may  have  been  somewhat  conservative;  however,  it  provided  a  baseline  data  set 
from  which  we  can  assess  how  well  the  estimated  location  works  to  predict  classification  quality 
in  the  live  site  environment. 

3.1.1  Data  Requirements 

In  order  to  assess  the  effectiveness  of  this  approach  we  compared  the  in-field  QC  decision  to  the 
results  obtained  using  the  expertise  of  an  off-site  analyst.  If  the  post-survey  analysis  identified 
any  initial  acquisition  files  that  should  have  been  reacquired,  but  were  not  identified  as  recollect 
opportunities  during  in-field  QC,  then  these  were  deemed  missed  opportunities. 

3.1.2  Metric 

The  actual  reacquisition  percentage  (Preac  -  total  number  of  reacquires  corresponding  to  missed 
recollect  opportunities  divided  by  total  number  of  possible  recollect  opportunities)  will  serve  as 
the  appropriate  metric.  The  number  of  missed  recollect  opportunities  corresponds  to  any  initial 
acquisition  files  that  were  identified  as  potential  recollect  opportunities  during  post-survey 
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analysis.  The  total  number  of  possible  recollect  opportunities  corresponds  to  the  number  of 
initial  acquisitions  where  no  subsequent  recollect  was  taken. 

3.1.3  Success  Criteria 

If  successful,  the  technology  should  significantly  reduce  reacquisition  rates.  We  will  consider  a 
reacquisition  percentage  PreaccO.  1  to  be  successful. 

3.2  OBJECTIVE:  EFFECTIVE  CORRECTIVE  ACTION 

Out  of  the  total  number  of  recollects  taken,  we  would  expect  that  in  most  of  these  instances,  the 
recollect  provides  better  characterization  of  the  target  than  the  initial  acquisition.  Therefore,  the 
percentage  of  recollects  resulting  in  improved  characterization  provides  a  good  measure  of  this 
objective. 

3.2.1  Data  Requirements 

To  determine  if  any  recollect  improves  characterization,  we  performed  both  qualitative  and 
quantitative  analyses  that  included  a  visual  QC  of  target  features  to  look  for  improvements  in 
resolution  as  well  as  an  evaluation  of  model  parameters  (e.g.,  see  section  3.6)  to  determine 
whether  the  recollect  provided  data  that  more  accurately  reflect  unique  target  features. 

3.2.2  Metric 

Prec(I),  the  number  of  recollects  that  provided  improved  characterization  divided  by  the  total 
number  of  recollects  taken,  serves  as  the  metric. 

3.2.3  Success  Criteria 

A  Prec(I)  value  greater  than  0.9  indicates  success. 

3.3  OBJECTIVE:  EFFECTIVE  IN-FIELD  CHARACTERIZATION 

Some  of  the  cases  corresponding  to  ineffective  recollects  were  likely  a  result  of  inadequate  in¬ 
field  model  capabilities.  For  example,  these  limitations  could  be  a  result  of  applying  a  single 
source  solver  when  multiple  sources  are  present,  or  an  inability  to  mask  out  ground  effects  from 
magnetic  soil. 

3.3.1  Data  Requirements 

For  the  purpose  of  future  improvements  to  the  in-field  QC  process,  we  wanted  to  identify  cases 
where  the  in-field  modeling  capabilities  may  limit  the  technology  performance.  This  analysis 
included  a  review  of  all  data  files  associated  with  ineffective  recollects  (identified  from  metric 
3.2.2).  For  these  cases  we  determined  the  cause  of  the  ineffective  recollect,  such  as  multiple 
sources  or  magnetic  soil  influence. 

3.3.2  Metric 

We  identified  the  total  number  of  cases  where  the  in-field  model  produced  inadequate 
characterization  of  the  target  space.  This  number  divided  by  the  total  number  of  recollects  serves 
as  the  metric  to  gauge  performance. 
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3.3.3  Success  Criteria 

If  the  total  number  of  cases  corresponding  to  inadequate  characterization  is  less  than  5%  of  the 
total  number  of  recollects,  the  performance  objective  will  be  met. 

3.4  OBJECTIVE:  EFFECTIVE  QUALITY  METRICS 

Any  additional  ineffective  recollects  could  correspond  to  cases  where  a  recollect  was 
unnecessary  due  to  inherent  sensor  limitations  (i.e.,  repositioning  the  sensor  was  ineffective 
because  the  target  was  beyond  the  classification  capabilities  of  the  sensor)  or  because  the  initial 
acquisition  provided  sufficient  characterization  of  target  features.  These  are  cases  where  the 
recollect  may  have  been  unnecessary.  Such  cases  are  indicative  of  ineffective  use  of  quality 
metrics. 

3.4.1  Data  Requirements 

For  this  analysis,  we  applied  data  and  model  parameters  (besides  estimated  target  location)  to 
these  data  to  determine  if  these  recollects  could  have  been  avoided  by  using  additional  quality 
metrics.  We  looked  at  several  parameters  including  model  fit,  model  noise,  data  noise,  etc.  that 
could  be  applied  during  the  in-field  process  to  potentially  reduce  unnecessary  recollects  (i.e., 
cases  where  the  recollect  cannot  improve  target  characterization)  without  missing  any  necessary 
recollect  opportunities. 

3.4.2  Metric 

The  total  number  of  unnecessary  recollects  that  could  be  averted  by  using  additional  data  or 
model  parameters  to  serve  as  quality  metrics  is  the  metric  for  this  objective. 

3.4.3  Success  Criteria 

Any  reduction  in  unnecessary  recollects  without  an  increase  in  missed  recollect  opportunities 
will  identify  effective  quality  metrics. 

3.5  OBJECTIVE:  PRODUCTION  RATE 

Production  rate  is  a  good  indicator  of  the  overall  efficiency  gains  that  are  achieved  by  including 
the  QC  process  in  the  survey  protocol.  While  production  rate  is  dependent  on  a  number  of 
factors  including  site  conditions,  we  can  make  some  assumptions  that  will  enable  a  quantitative 
comparison  of  production  rates  achieved  with  and  without  the  QC  process. 

3.5.1  Data  Requirements 

For  each  day  of  the  survey,  we  analyzed  production  rates  corresponding  to  a  sample  of 
consecutive  data  files.  These  samples  were  chosen  to  reflect  ideal  working  conditions  when 
production  was  uninterrupted  by  things  like  trips  to  the  IVS,  replacement  of  batteries,  equipment 
down  time,  equipment  transport  across  the  site,  etc.  Therefore  these  samples  tended  to  reflect  the 
“optimal”  production  rate  rather  than  the  rate  actually  achieved  over  the  course  of  each  day.  We 
estimated  this  optimal  production  rate  as  a  way  to  understand  how  the  QC  module  might  affect 
production  under  ideal  working  conditions,  and  therefore,  this  production  rate,  to  a  large  extent, 
did  not  reflect  working  conditions  specific  to  the  WMA  site.  Our  optimal  production  rate  was 
estimated  by  analyzing  the  time  stamp  information  from  a  group  of  consecutive  files  collected 
over  a  period  each  day  when  production  was  uninterrupted  (at  least  an  hour  or  more  for  each 
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group  of  files).  Each  file  is  time-stamped  with  a  Coordinated  Universal  Time  (UTC)  value. 
Based  on  the  time-stamp  information,  we  were  able  to  establish  production  rate  metrics 
corresponding  to  both  the  QC  process  and  the  standard  protocol.  These  production  rate  metrics 
were  averaged  over  several  days  to  establish  the  representative  ideal  production  rates. 

3.5.2  Metric 

For  the  production  rate  corresponding  to  the  QC  process,  we  obtained  an  average  number  of 
unique  anomalies  (i.e.,  recollects  not  included)  interrogated  per  hour.  While  the  recollects  were 
not  included  in  this  number,  we  did  include  the  time  required  to  perform  these  recollects.  Thus, 
for  example,  if  it  took  the  field  team  2  hours  to  collect  100  files,  but  only  70  of  these  were  initial 
acquisitions  (i.e.,  30  recollects)  the  baseline  production  rate  would  be  35/hr.  For  the  production 
rate  corresponding  to  the  standard  protocol  (i.e.,  no  QC  process  included),  we  subtracted  the  time 
required  to  perform  the  recollects  from  the  calculation  (based  on  UTC  time  stamp).  Thus,  out  of 
the  100  original  files  acquired  in  2  hours,  if  it  took  the  field  team  1  hour  40  minutes  to  perform 
the  70  initial  acquisitions,  this  would  result  in  a  baseline  production  rate  of  42/hr. 

Finally,  to  establish  a  true  production  rate,  we  divided  the  baseline  rate  by  a  reacquisition  factor. 
The  reacquisition  factor  was  determined  based  on  an  assessment  of  the  survey  totals.  For 
example,  this  survey  included  1032  unique  anomalies  with  46  necessary  recollects  and  1  possible 
reacquisition  (based  on  objective  3.1  and  3.2  analyses).  Thus,  the  reacquisition  factors  were 
(1032+1)/1032=1.001  and  (1032+46)/1032=1.045  for  the  QC  rate  and  the  standard  rate, 
respectively.  Thus  in  our  example,  the  true  production  rates  would  be  35/1.001=35.0  and 
42/1 .045=40.2  for  the  QC  process  and  standard  protocol,  respectively. 

3.5.3  Success  Criteria 

Any  increase  in  the  production  rate  corresponding  to  the  incorporation  of  the  QC  process  will  be 
deemed  a  success. 

3.6  Quantifying  Improvements  in  Target  Features 

Several  of  the  aforementioned  objectives  (specifically  objectives  3.2,  3.3,  and  3.4)  require 
quantification  of  improvements  (or  lack  thereof)  in  target  classification  features.  The  most 
straightforward  method  to  quantify  improvements  in  classification  features  is  to  apply  a  fit 
metric  to  known  library  features.  While  this  method  is  applicable  to  targets  of  interest  for  which 
libraries  exist,  many  of  the  recollects  corresponded  to  non-TOIs  and  therefore  required 
application  of  different  metrics  to  quantify  feature  improvements. 

For  non-TOIs  (or  targets  for  which  libraries  do  not  exist)  we  applied  two  feature  quality  metrics: 

1.  Model  Noise  -  a  measurement  of  point-to-point  jitter  within  the  recovered  principal 
polarizabilities;  a  value  lower  than  0.1  for  the  ratio  between  the  recollect  model  noise 
standard  deviation  and  the  initial  acquisition  model  noise  standard  deviation  indicated  an 
improvement  in  target  features. 

2.  Model  Consistency  -  a  measurement  of  consistency  between  principal  polarizabilities 
recovered  from  the  initial  acquisition  and  those  recovered  from  the  recollect;  a  deviation 
of  greater  than  10%  in  any  polarizability  match  between  the  initial  acquisition  data  and 
the  recollect  data  indicated  an  improvement  in  target  features. 
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The  following  examples  demonstrate  the  application  of  these  metrics.  For  each  of  these 
examples,  we  show  the  analysis  results  corresponding  to  data  collected  over  targets  for  which 
libraries  exist;  thus,  we  have  a  well  defined  performance  baseline  reference  to  which  we  can 
compare  the  decisions  based  on  the  aforementioned  feature  quality  metrics. 


The  first  example  shows  inversion  results  for  data  collected  over  a  60mm  mortar  at  30cm  burial 
depth  (Figure  6).  The  initial  data  file  was  acquired  with  the  MetalMapper  center  offset 
approximately  55cm  from  the  target  center;  the  recollect  was  acquired  with  the  MetalMapper 
center  approximately  aligned  with  the  target  center.  The  recollect  data  clearly  provide  better 
target  features  as  the  polarizabilities  generated  from  inversion  of  the  recollect  data  show  a  much 
closer  match  to  the  library  polarizabilities.  Specifically,  the  L3  polarizability  (green  line) 
corresponding  to  the  initial  data  shows  a  significant  deviation  from  the  library  in  late  time. 
Additionally,  the  L2  polarizability  (red  line)  corresponding  to  the  recollect  data  shows  a  slightly 
better  match  to  the  library  over  the  complete  decay  period. 


Time  (s) 


Time  (s) 


Figure  6.  LEFT:  Polarizabilities  recovered  from  inversion  of  initial  data  acquired  with  55cm  offset  from 
target.  RIGHT:  Polarizabilities  recovered  from  inversion  of  recollect  data  acquired  with  MetalMapper 
centered  over  target.  The  blue,  red,  and  green  lines  represent  the  primary,  secondary,  and  tertiary  principal 
polarizabilities,  respectively  (LI,  L2,  and  L3,  respectively).  The  grey  lines  represent  the  principal 
polarizabilities  for  the  reference  library  match  (60mm  mortar). 

With  reference  libraries,  it  is  relatively  easy  to  identify  improvements  in  the  target  features.  To 
understand  how  the  feature  quality  metrics  perform  in  the  absence  of  libraries,  we  can  apply  the 
aforementioned  metrics  to  both  data  sets  (Table  2). 


Table  2.  Feature  Quality  Metrics  for  60mm  Mortar 


POLARIZABILITY 

FEATURE  QUALITY  METRIC 

Model  Noise  (<0.1) 

Model  Consistency  %  (>10%) 

LI 

1.44 

1.46 

L2 

0.86 

L3 
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Feature  quality  metric  values  in  Table  2  that  meet  the  objective  thresholds  (<0.1  and  >10%  for 
the  model  noise  and  model  consistency  metrics,  respectively)  are  highlighted  in  red.  These 
values  indicate  improvements  in  feature  quality.  The  values  in  Table  2  can  be  related  to  the 
feature  library  matches  shown  in  Figure  6.  The  feature  quality  metric  values  do  not  indicate  any 
significant  improvement  in  the  LI  polarizability  and  this  is  confirmed  by  the  library  match.  The 
model  consistency  metric  indicates  a  slight  improvement  (13.2%  compared  to  10%  threshold)  in 
the  L2  polarizability  and  this  is  supported  by  the  marginally  better  library  match  for  the  recollect 
L2.  Finally,  both  the  model  noise  and  model  consistency  metrics  indicate  significant 
improvements  in  the  L3  polarizability.  These  improvements  are  evident  in  the  L3  library  match, 
which  shows  significant  jitter  in  late  time  (i.e.,  model  noise)  as  well  as  significant  deviation  from 
the  library  polarizability  throughout  the  decay  period  (i.e.,  model  consistency)  for  the  initial 
acquisition. 

For  reference  purposes,  the  model  noise  values  corresponding  to  the  initial  and  recollect  files  are 
shown  in  Figure  7.  Typically,  only  noise  values  out  to  2.8  ms  are  used  in  the  standard  deviation 
calculations  since  later  time  gates  can  have  a  spurious  influence. 


'3)  05  i  fs  2  2.5  3 

Time  (s)  x  -iq'3 
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Figure  7.  Model  noise  values  for  LI  (top),  L2  (middle),  and  L3  (bottom).  Dashed  lines  represent  noise  values 
associated  with  the  initial  data  file;  solid  lines  represent  noise  values  associated  with  the  recollect  data  file.  It 
is  clear  that  only  the  L3  shows  any  significant  improvements  in  noise  associated  with  the  recollect. 

The  next  example  demonstrates  how  the  feature  quality  metrics  can  be  applied  to  identify  a  case 
where  a  recollect  is  unnecessary.  Figure  8  shows  the  analysis  results  for  data  collected  over  a 
QC  seed  during  a  production  MetalMapper  survey.  The  seed  object  was  a  small  ISO40.  The 
figure  shows  the  polarizabilities  recovered  from  the  data  corresponding  to  both  the  initial 
acquisition  and  the  recollect.  During  the  initial  acquisition,  the  MetalMapper  center  was  offset 
more  than  35  cm  laterally  from  the  ISO;  during  the  recollect,  the  MetalMapper  was 
approximately  centered  over  the  ISO. 


Figure  8.  LEFT:  Polarizabilities  recovered  from  inversion  of  initial  data  acquired  with  >35cm  offset  from 
target.  RIGHT:  Polarizabilities  recovered  from  inversion  of  recollect  data  acquired  with  MetalMapper 
centered  over  target.  The  blue,  red,  and  green  lines  represent  the  primary,  secondary,  and  tertiary  principal 
polarizabilities,  respectively  (LI,  L2,  and  L3,  respectively).  The  grey  lines  represent  the  principal 
polarizabilities  for  the  reference  library  match  (small  ISO40). 
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Visual  inspection  of  the  recovered  polarizabilities  does  not  indicate  a  clear  improvement  in  the 
features  recovered  from  the  recollect.  The  L2  and  L3  polarizabilities  appear  slightly  noisier  in 
the  initial  acquisition;  however,  in  both  cases  a  correct  classification  decision  was  made.  A 
similar  conclusion  can  be  made  by  using  the  feature  quality  metrics  instead  of  the  target  libraries. 
Feature  quality  metric  values  are  shown  in  Table  3. 


Table  3.  Feature  Quality  Metrics  for  Small  ISO40 


POLARIZABILITY 

FEATURE  QUALITY  METRIC 

Model  Noise  (<0.1) 

Model  Consistency  %  (>10%) 

LI 

1.84 

0.06 

L2 

0.22 

3.22 

L3 

0.15 

6.38 

In  this  example,  values  for  both  the  model  noise  and  model  consistency  metrics  are  outside  the 
objective  thresholds,  indicating  no  significant  improvement  in  the  classification  features.  In  this 
case,  the  recollect  was  unnecessary.  This  conclusion  is  also  supported  by  the  visual  inspection. 

It  should  be  noted  that  although  the  model  consistency  metric  does  not  provide  a  direct 
measurement  of  the  polarizability  match  to  a  library  value  it  generally  correlates  strongly  with 
library  match  quality.  The  reason  for  this  correlation  is  because  the  recollect  will  reliably 
produce  a  very  close  library  match  assuming  the  sensor  is  properly  centered  over  the  object  (also 
assuming  no  significant  influence  from  other  sources).  The  initial  acquisition  may  or  may  not 
produce  a  close  library  match  depending  on  the  lateral  offset  and  depth  of  the  object.  Thus,  the 
model  consistency  metric  applies  the  assumption  that  the  recollect  will  provide  good 
characterization  of  the  object  and  therefore  any  significant  deviations  between  initial  acquisition 
results  and  the  recollect  results  are  indicative  of  improvements  in  target  features.  Again,  this 
metric  is  only  valid  if  we  can  ensure  that  the  recollect  provides  good  characterization  of  the 
object. 

Consequently,  we  must  also  apply  secondary  metrics  such  as  model  fit  to  confirm  the  reliability 
of  the  recollect  (such  metrics  play  an  important  role  in  analysis  related  to  objective  3.3:  Effective 
In-Field  Characterization).  In  scenarios  where  multiple  objects  are  present,  the  model  fit  metric 
(particularly  when  used  in  conjunction  with  joint  diagonalization  analysis)  is  a  reliable  indicator 
of  multiple  sources.  The  following  example  demonstrates  the  effectiveness  of  applying  model 
fit. 

Figure  9  shows  an  example  of  a  multi-target  scenario.  In  this  case,  the  MetalMapper  was 
centered  over  a  large  piece  of  frag  while  a  60mm  mortar  was  located  at  a  large  lateral  offset  from 
the  center  of  the  sensor.  Applying  a  single- source  solver  does  not  produce  a  close  match  to  the 
60mm  library;  however,  when  a  two-source  solver  is  applied,  a  resulting  set  of  polarizabilities  is 
produced  that  shows  a  good  match  to  the  60mm  library  (Figure  10). 
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Figure  9.  Test  pit  layout  for  multi-object  scenario.  The  MetalMapper  center  was  positioned  over  the  frag 
item. 


Figure  10.  LEFT:  Single-source  results  for  the  multi-object  data  collection.  CENTER  and  RIGHT:  Two- 
source  results  for  multi-object  data  collection.  Colored  lines  represent  the  recovered  LI,  L2,  and  L3 
polarizabilities  (blue,  red,  and  green,  respectively).  Grey  lines  represent  the  60mm  library  polarizabilities. 
Target  2  (right)  shows  a  good  match  to  the  60mm  library. 

The  improved  target  space  characterization  provided  by  the  two-source  solver  can  be  quantified 
by  applying  the  model  fit  metric.  The  residual  error  between  the  data  and  the  model  output  is 
significantly  lower  for  the  results  of  the  two-source  solution,  indicating  that  the  two-source 
model  provides  a  better  representation  of  the  target  space.  Figure  11  and  Figure  12  show 
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data/model  comparisons  for  the  single-source  and  two-source  analyses,  respectively.  It  is 
evident  that  the  two-source  solver  results  in  a  better  model  fit. 

i  a*10'* — , - , - - - - t - T - = 

— data 
— model 

1.6b  - A 


Samples 

Figure  11.  Single-source  model  fit  comparing  concatenated  data  (blue)  and  model  (red)  outputs. 


Figure  12.  Two-source  model  fit  comparing  concatenated  data  (blue)  and  model  (red)  outputs. 

While  model  fit  is  useful  for  identifying  potential  multi-object  cases,  it  does  not  necessarily  serve 
as  a  good  metric  for  determining  feature  quality.  In  fact,  in  most  cases  model  fit  does  not 
effectively  identify  improvements  in  target  classification  features.  Consider  the  following 
example  data  set  collected  over  a  37mm  projectile  (Figure  13). 
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Figure  13.  LEFT:  Polarizabilities  recovered  from  inversion  of  initial  data  acquired  with  >35cm  offset  from 
target.  RIGHT:  Polarizabilities  recovered  from  inversion  of  recollect  data  acquired  with  MetalMapper 
centered  over  target.  The  blue,  red,  and  green  lines  represent  the  primary,  secondary,  and  tertiary  principal 
polarizabilities,  respectively  (LI,  L2,  and  L3,  respectively).  The  grey  lines  represent  the  principal 
polarizabilities  for  the  reference  library  match  (37mm  projectile). 

The  recollect  clearly  results  in  improved  classification  features;  however,  this  result  is  not 
supported  by  the  model  fit  metric  (Figure  14  and  Figure  15).  Both  the  initial  acquisition  and  the 
recollect  result  in  good  model  fits  (in  fact,  the  residual  error  between  the  model  and  data  is 
slightly  lower  for  the  initial  acquisition).  The  initial  acquisition  provides  a  good  model  fit; 
however,  the  data  do  not  sufficiently  constrain  the  inversion  to  produce  a  unique  solution  that 
matches  the  true  polarizabilities. 


Figure  14.  Initial  acquisition  model  fit  comparing  concatenated  data  (blue)  and  model  (red)  outputs.  The 
data  produce  a  good  model  fit,  but  the  solution  is  non-unique. 
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Figure  15.  Recollect  model  fit  comparing  concatenated  data  (blue)  and  model  (red)  outputs.  The  data 
produce  a  good  model  fit  and  a  unique  solution. 

Thus,  the  model  fit  should  be  used  only  to  support  decisions  that  are  based  primarily  on  the 
feature  quality  metrics.  By  applying  these  feature  quality  metrics  retrospectively  to  the 
demonstration  data  set,  we  have  a  quantifiable  way  to  identify  instances  where  target  features 
were  improved  as  well  as  instances  where  recollects  were  unnecessary. 

4.  SITE  DESCRIPTION 

MetalMapper  data  collected  at  the  Former  Waikoloa  Maneuver  Area  provide  the  basis  for  this 
technology  demonstration.  This  area  comprises  approximately  100,000  acres  on  the  northwest 
portion  of  the  island  of  Hawaii  and  includes  property  that  was  used  as  a  military  training  camp 
and  artillery  range  (Figure  16).  Over  100  munitions  types  including  mortars,  projectiles,  hand 
grenades,  and  rockets  are  known  to  be  present  at  the  site.  Details  regarding  site  history  and  land 
usage  can  be  found  on  the  USACE  Honolulu  District  web  site: 

http://www.poh.usace.armv.mil/Missions/Environmental/FUDS/Waikoloa.aspx 
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Figure  16.  Former  Waikoloa  Maneuver  Area  (from  USACE  Waikoloa  map). 

The  arid  site  contains  sparse  vegetation  making  it  suitable  for  vehicle  surveys  (Figure  17).  The 
main  deployment  challenge  is  the  rocky  soil,  which  can  damage  tires  and  make  transport  of 
equipment  across  the  site  difficult.  The  site  is  also  known  to  contain  significant  magnetic 
geology,  which  can  be  problematic  for  electromagnetic  background  removal  during  data 
processing. 


Figure  17.  Former  Waikoloa  Maneuver  Area  site  photos. 
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MetalMapper  data  were  collected  within  three  areas  of  interest  (AOI),  which  included  task  order 
(TO)  17  and  TO20  areas  A  and  B.  Each  of  these  areas  provided  unique  magnetic  geology  that 
created  variable  levels  of  electromagnetic  background  across  the  data  set.  The  magnetic  soil 
influence  proved  to  be  one  of  the  most  significant  challenges  in  the  implementation  of 
classification  and  in-field  QC  at  this  site. 

5.  TEST  DESIGN 

Demonstration  of  the  in-field  QC  approach  was  included  as  a  supplement  to  ongoing 
classification  studies  conducted  at  the  WMA.  The  unique  geology  of  the  WMA  offered 
significant  challenges  to  classification  and  therefore,  this  site  was  selected  for  demonstration  of 
several  ESTCP  technologies.  We  coordinated  with  Parsons,  who  were  conducting  one  of  the 
classification  technology  demonstrations  at  this  site,  to  provide  their  field  team  a  version  of  the 
in-field  QC  module  to  use  during  their  cued  MetalMapper  survey  of  the  aforementioned  three 
AOI.  Some  of  the  Parsons  field  personnel  were  familiar  with  or  had  prior  experience  using  the 
in-field  QC  software  so  transition  of  the  technology  was  relatively  straightforward. 

Our  objective  for  conducting  this  demonstration  was  to  obtain  data  about  the  utility  of  the  in¬ 
field  QC  process  and  at  the  same  time  to,  hopefully,  provide  the  Parsons  team  with  a  technology 
that  would  facilitate  the  demonstration  of  their  technology  as  well.  In  order  to  minimize 
redundancy  in  the  reporting  of  these  related,  but  parallel  efforts,  we  will  not  include  details  about 
the  WMA  MetalMapper  demonstration  field  procedures  in  this  document.  A  comprehensive 
summary  of  details  related  to  the  field  activities  (e.g.,  system  calibration,  data  collection,  ground 
truth  validation,  etc.)  for  this  demonstration  can  be  found  in  the  Parsons  MR-201104 
demonstration  report  (see  Van  et  al.,  2015). 

6.  DATA  ANALYSIS  PLAN 

Our  analysis  focused  on  retrospective  interpretation  of  classification  results  to  determine  the 
effectiveness  of  the  in-field  QC  process.  To  obtain  relevant  data  and  model  parameters  from 
each  MetalMapper  anomaly,  we  performed  dipole  inversions  (single  and  multi-object)  for  each 
static  MetalMapper  file.  These  inversions  provided  parameters  such  as  polarizabilities, 
estimated  location,  and  fit  that  we  subsequently  used  to  obtain  metrics  for  each  anomaly.  We 
also  coordinated  with  analysts  from  Dartmouth  College  (MR-201227,  PI:  Shubitidze),  who 
performed  primary  classification  analysis  for  the  WMA  site,  to  identify  any  specific  anomalies 
that  might  further  elucidate  the  benefits  or  shortcomings  of  the  in-field  QC  method. 

6.1  PARAMETER  ESTIMATION 

To  obtain  the  relevant  parameters  that  would  quantify  any  improvements  in  recollected  data  files, 
we  performed  dipole  inversions  for  each  MetalMapper  file.  One  of  the  most  important  steps  in 
this  processing  stage  was  the  selection  of  background  data  that  would  be  used  for  background 
removal  in  each  cued  data  file.  Because  background  response  was  so  variable  across  the  WMA, 
selection  of  different  backgrounds  could  produce  very  different  sets  of  model  parameters 
obtained  from  each  inversion.  We  initially  used  background  files  that  had  timestamps  closest  to 
those  of  the  associated  cued  files.  If  background  removal  using  these  files  still  left  a  significant 
background  response  in  the  data,  we  would  try  other  background  files  collected  on  the  given  day 
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to  see  if  any  improvements  in  background  removal  could  be  realized.  In  some  cases,  it  was  only 
possible  to  obtain  parameters  associated  with  a  background  response  (this  was  the  case,  for 
example,  with  what  turned  out  to  be  “no-contact”  sources). 

After  background  removal,  we  inverted  each  static  file  using  both  single  and  multi-object  dipole 
models.  In  some  cases,  the  multi-object  could  be  used  to  constrain  the  ground  response.  The 
polarizabilities,  estimated  source  location,  and  data/model  fit  for  each  inversion  were  saved  for 
further  review  of  performance  metrics. 

6.2  PERFORMANCE  ANALYSIS 

Performance  analysis  included  a  review  of  the  aforementioned  parameters  associated  with  each 
recollect  anomaly  as  well  as  identification  of  specific  anomalies  that  proved  challenging  for  the 
final  classification  stage  (i.e.,  based  on  retrospective  ROC  curve  analysis).  We  applied  the 
objective  metrics  to  each  of  these  cases. 

6.2.1  Objective  3.1:  Identification  of  All  Recollect  Opportunities 

Any  cued  sounding  based  on  a  DGM  pick  could  be  considered  a  recollect  opportunity;  however, 
one  of  the  biggest  issues  with  identifying  these  opportunities  was  whether  to  include  the  large 
number  of  no-contact  encounters  (i.e.,  no  target  found  during  excavation)  as  a  potential  recollect 
opportunity.  There  were  a  large  number  anomalies  (about  half  of  all  the  cued  encounters)  that 
could  not  be  attributed  to  a  legitimate  source  (other  than  the  ground  or  “hot  rocks”).  Therefore, 
these  encounters  would  have  a  significant  impact  on  the  overall  recollect  statistics.  Because 
these  encounters  represented  an  opportunity  to  make  a  “no  recollect”  decision,  we  included  them 
in  the  analysis;  however,  it  should  be  noted  that  the  in-field  QC  module  did  not  have  specific 
models  to  reject  ground  response,  therefore,  we  would  expect  such  cases  to  be  challenging  for 
the  in-field  decision. 

6.2.2  Objective  3.2:  Effective  Corrective  Action 

To  determine  whether  effective  corrective  action  was  taken,  we  compared  the  recovered  model 
parameters  from  each  recollect  to  those  from  the  initial  acquisition.  For  recollects  associated 
with  TOI,  this  analysis  applied  a  library  fit  to  the  polarizabilities  to  determine  any  improvements 
in  target  features.  For  non-TOI  encounters,  we  applied  the  model  parameters  described  in 
section  3.6  to  determine  if  the  recollect  provided  improvements  in  features.  Finally,  for  cases 
where  we  observed  strong  ground  response,  we  assessed  the  polarizability  decay  to  determine 
whether  the  features  were  associated  with  a  metal  object  or  the  ground  response. 

6.2.3  Objective  3.3:  Effective  In-Field  Characterization 

For  each  recollect  case  that  did  not  lead  to  improved  target  features,  we  identified  the  reason  that 
classification  quality  was  not  improved.  Because  we  were  interested  particularly  in  cases  where 
the  in-field  models  did  not  accurately  characterize  the  target  space,  we  evaluated  data/model  fits 
and  ground  truth  records  to  determine  if,  for  example,  multiple  targets  in  the  area  may  have  led 
to  misguided  placement  of  the  sensor.  We  also  assessed  the  polarizability  decay  to  determine  if 
ground  response  likely  had  any  significant  influence  on  the  in-field  decision. 
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6.2.4  Objective  3.4:  Effective  Quality  Metrics 

For  recollect  cases  where  the  initial  acquisition  provided  sufficient  characterization  of  the 
anomaly,  we  evaluated  the  application  of  alternative  metrics  to  support  a  recollect  decision.  We 
also  evaluated  whether  the  positioning  requirements  (30  cm  or  closer)  for  the  initial  acquisition 
were  overly  stringent  and  resulted  in  too  many  recollects.  Our  goal  was  to  determine  if  these 
unnecessary  recollects  could  be  avoided  without  risking  any  reduction  in  data  quality. 

6.2.5  Objective  3.5:  Production  Rate 

To  establish  any  efficiency  gains  resulting  from  the  in-field  QC  process,  we  evaluated  the 
production  rates  corresponding  to  the  in-field  QC  protocol  and  the  standard  data  collection 
protocol  (no  in-field  QC)  as  described  in  section  3.5.2.  We  used  the  data  time  stamps  in  several 
groups  of  consecutively  recorded  MetalMapper  files  to  establish  the  average,  ideal  production 
rates  for  both  data  collection  methods. 

7.  PERFORMANCE  ASSESSMENT 

The  following  is  a  summary  of  the  recollect  statistics  for  the  WMA  MetalMapper  survey: 

•  Out  of  1032  total  anomaly  encounters,  there  were  231  unique  recollects; 

•  Out  of  104  TOI  encounters,  there  were  18  unique  recollects; 

•  Out  of  393  non-TOI  (debris)  encounters,  there  were  99  unique  recollects; 

•  Out  of  535  no-contact  (no  target)  encounters,  there  were  1 14  unique  recollects. 

These  statistics  provide  the  basis  for  our  analysis  of  the  performance  objectives. 

7.1  Objective  3.1:  Identification  of  All  Recollect  Opportunities 

To  determine  if  there  were  any  missed  recollect  opportunities  during  the  WMA  survey,  we 
identified  cases  where  the  final  MetalMapper  location  was  more  than  30  cm  from  an  excavated 
source.  We  found  three  potential  cases. 

The  first  case  involved  anomaly  WK-73,  a  37mm  seed  buried  at  11  cm  depth.  In  this  case,  the 
initial  (and  only)  MetalMapper  file  associated  with  this  anomaly  was  acquired  at  an  offset  of  75 
cm.  This  anomaly  was  problematic  for  analysts  because  the  only  features  obtained  were 
associated  with  a  ground  response  due  to  the  large  offset.  While  repositioning  over  the  seed 
location  would  undoubtedly  have  provided  better  data  for  characterizing  the  anomaly,  we  do  not 
feel  this  was  an  instance  where  the  in-field  QC  failed.  A  75  cm  offset  placed  the  seed  well 
outside  the  footprint  of  the  sensor.  Under  ideal  conditions  (i.e.,  no  significant  soil  response  or 
presence  of  other  nearby  anomalies)  it  is  possible  that  the  in-field  model  could  have  located  a 
source  this  far  from  the  sensor  and  provided  some  guidance  for  better  positioning;  however, 
given  the  challenges  posed  by  field  conditions  we  would  not  typically  expect  accurate  features 
would  be  obtained  from  a  source  located  this  far  from  the  sensor.  This  case  represented  an  error 
in  the  initial  positioning  of  the  sensor  and  was  therefore  removed  from  the  list  for  classification 
analysis. 
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The  other  two  cases  involved  anomalies  WK-1027  and  WK1047,  which  were  a  small  ISO  seed 
and  a  60mm  seed,  respectively.  These  two  seeds  proved  problematic  for  the  classification 
analysts  as  evidenced  by  the  ROC  curve  shown  in  Figure  18. 


MM 


Number  of  Non-TOIs  Incorrectly  Classified 


Figure  18.  Independent  scoring  results  for  Dartmouth  College  WMA  MetalMapper  analysis  indicate  two 
difficult  targets:  WK-1027  and  WK-1047. 

In  both  of  these  cases,  the  features  obtained  during  analysis  appeared  to  be  dominated  or 
degraded  by  the  ground  response.  Thus,  insufficient  background  removal  appeared  to  be  the 
major  cause  of  the  classification  failure;  however,  we  wanted  to  determine  if  the  sensor 
positioning  could  have  been  a  factor  as  well.  Since  neither  of  these  cases  included  a  recollect,  we 
assessed  whether  the  initial  sensor  location  may  have  contributed  to  the  incorrect  classification 
decision. 

For  seed  WK-1027,  the  sensor  was  located  27  cm  from  the  target.  Since  this  offset  is  within  the 
conservative  30  cm  objective  radius,  we  do  not  consider  this  case  to  be  a  failure  of  the  in-field 
QC  model.  It  is  possible  that  given  the  small  size  of  the  object  and  the  dominant  ground 
response,  that  positioning  the  center  of  the  sensor  closer  to  the  object  could  have  provided  better 
results  in  this  case;  however,  we  believe  this  case  is  more  representative  of  the  challenges  posed 
by  magnetic  soils  than  of  the  limitations  of  the  in-field  QC  approach. 

For  seed  WK-1047,  the  sensor  was  located  43  cm  from  the  target.  This  offset  is  significantly 
outside  the  30  cm  objective  radius  and,  therefore,  we  consider  this  to  be  a  missed  recollect 
opportunity.  While  it  is  likely  that  under  ideal  conditions  (i.e.,  no  significant  soil  response  or 
presence  of  other  nearby  anomalies)  it  would  be  possible  to  obtain  accurate  classification 
features  at  a  43  cm  offset,  the  presence  of  a  significant  ground  response  makes  it  very  difficult  to 
obtain  representative  features  with  the  sensor  at  this  location.  Most  likely,  if  the  in-field  QC 
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model  had  access  to  a  representative  background  at  this  location,  a  better  QC  decision  could  have 
been  made  at  the  time. 

To  obtain  our  metric  for  this  objective,  we  identified  the  number  of  remaining  recollect 
opportunities  (1032  -  231  =  801).  Out  of  801  possible  additional  recollect  opportunities,  we 
believe  there  is  only  1  case  where  repositioning  the  sensor  may  have  led  to  a  better  classification 
result.  Therefore  the  PreaccO.l  was  achieved. 

7.2  Objective  3.2:  Effective  Corrective  Action 

To  determine  whether  the  recollects  provided  significant  improvements  in  the  classification 
quality  of  the  data,  we  analyzed  the  features  obtained  from  each  recollect  and  compared  them  to 
those  of  the  associated  initial  acquisition.  For  recollects  over  TOI,  improvements  were 
quantified  by  assessing  the  polarizability  fit  to  the  relevant  TOI  library.  Figure  19  shows  an 
example  of  an  effective  recollect  over  a  37mm  seed. 


Figure  19.  Polarizabilities  obtained  from  two  different  acquisitions  over  WK-724,  a  37mm  seed.  For  the 
initial  acquisition,  the  sensor  was  offset  34  cm  from  the  target.  The  recollect  placed  the  sensor  at  an  offset  of 
13  cm  from  the  target.  The  polarizabilities  (blue,  red,  and  green  lines)  show  a  much  better  match  to  the 
library  (dark  grey  lines)  in  the  recollect. 

The  results  in  Figure  19  are  a  good  example  of  improvements  achieved  by  recollecting  with 
better  sensor  positioning  (13  cm  lateral  offset  in  the  recollect  compared  to  34  cm  lateral  offset  in 
the  initial  acquisition).  The  initial  acquisition  did  not  enable  accurate  recovery  of  the 
polarizabilities;  however,  the  recollect  provided  features  that  matched  closely  to  the  library. 
Review  of  the  data  showed  that  the  initial  acquisition  may  have  been  influenced  by  a  small, 
nearby  piece  of  clutter  that  was  less  evident  in  the  recollect  location. 
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For  recollects  over  non-TOI  (i.e.,  frag  or  other  debris)  we  quantified  any  significant 
improvements  using  the  model  parameters  described  in  section  3.6.  For  example,  Figure  20 
compares  the  polarizabilities  obtained  from  initial  and  recollect  acquisitions  over  munitions 
debris  (WK-1053,  fuze  clip). 
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Figure  20.  Polarizabilities  obtained  from  two  different  acquisitions  over  WK-1053,  a  piece  of  munitions 
debris.  The  recollect  provides  better  constraint  on  the  third  polarizability  (green  line). 


The  results  shown  in  Figure  20  were  quantified  using  the  model  noise  and  model  consistency 
metrics  described  in  section  3.6.  Further  validation  was  later  provided  by  library  matching  to 
polarizabilities  obtained  from  other  fuze  clips  encountered  throughout  the  site.  The  recollect 
shown  in  Figure  20  provided  features  that  were  much  more  consistent  with  those  from  other 
encounters  with  this  target,  which  turned  out  to  be  a  common  debris  item  at  the  site. 


Over  the  course  of  this  analysis,  we  discovered  several  encounters  where  the  recollect  provided 
an  entirely  different  set  of  features  from  those  of  the  initial  acquisition.  These  cases  occurred 
when  the  initial  acquisition  was  dominated  by  the  ground  response  and  the  recollect  was  able  to 
resolve  the  features  associated  with  the  actual  target.  A  visual  QC  check  of  the  polarizabilities 
could  identify  these  cases  as  shown  in  Figure  21. 


Overall,  we  found  46  cases  where  the  recollect  provided  significant  improvements  in  target 
features.  While  this  number  is  a  minority  of  the  231  total  recollects,  it  should  be  noted  that  most 
of  the  recollects  (153  cases)  appeared  to  be  due  to  inaccurate  source  locations  created  by  the 
ground  response.  Counting  these  ground  recollects  we  obtain  a  Prec(I)  =  0.19  (i.e.,  46  out  of  231 
recollects  resulted  in  improved  features),  and  disregarding  the  ground  recollects  we  obtain  a 
Prec(I)  =  0.59  (i.e.,  46  out  of  78  recollects  provided  improvements).  Both  of  these  values  fall 
short  of  the  objective  Prec(I)  =  0.9  (i.e.,  90%  of  recollects  result  in  improved  target  features). 
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Figure  21.  Example  of  the  recollect  providing  better  resolution  of  target  features.  In  this  case  the  initial 
acquisition  produced  polarizabilities  dominated  by  the  ground  response.  The  recollect  provides  features  that 
more  accurately  represent  the  actual  target  (a  piece  of  frag). 

7.3  Objective  3.3:  Effective  In-Field  Characterization 

The  majority  of  cases  where  the  recollect  did  not  lead  to  improved  target  features  appeared  to  be 
a  result  of  insufficient  background  removal  or  inadequate  ground  models.  An  example  of  this  is 
provided  in  Figure  22,  which  shows  the  initial  acquisition  and  the  recollect  over  anomaly  WK- 
68,  a  piece  of  frag. 

Multiple  objects  did  not  appear  to  present  any  significant  problems  for  the  in-field  QC.  We  did 
not  find  any  cases  where  an  absence  of  multi-source  solvers  in  the  in-field  models  led  to  a  poor 
classification  of  a  target.  While  there  were  several  instances  where  multiple  sources  were  found 
within  the  search  area,  most  of  these  cases  included  a  number  of  small  debris  items.  For  TOI  as 
small  as  20mm,  it  is  possible  these  cases  could  have  been  problematic;  however,  the  smallest 
TOI  encountered  during  the  survey  was  a  37mm,  which  was  not  small  enough  to  be  masked  by 
any  of  the  nearby  debris  items. 

Out  of  the  231  recollects,  we  found  153  cases  where  the  soil  response  appeared  to  influence  the 
in-field  QC  decision,  resulting  in  insufficient  characterization  of  the  area.  This  number  is 
significantly  greater  than  the  objective  5%  of  total  recollects;  however,  given  the  known 
challenges  associated  with  the  site’s  geology,  and  the  absence  of  any  methods  to  mitigate  these 
effects  in  the  in-field  software  these  results  are  reasonable. 


White  River  Technologies 


31 


January  2016 


ESTCP -MR-201264  Real  Time  Quality  Control  Methods  for  Cued  EMI  Data  Collection 


Si 

m 

re 

o 

CL 


10 

Time  (s) 


10 

Time  (s) 


Initial: 


68 


Figure  22.  Polarizabilities  associated  with  anomaly  WK-68,  a  piece  of  frag.  Here  the  initial  acquisition  shows 
good  resolution  of  target  features;  however,  the  recollect  provides  features  dominated  by  the  ground 
response. 

7.4  Objective  3.4:  Effective  Quality  Metrics 

The  remainder  of  ineffective  recollects  (i.e.,  recollects  that  did  not  improve  characterization  and 
that  were  not  a  result  of  magnetic  geology  or  multiple  target  influence)  provided  an  opportunity 
to  test  quality  metrics  other  than  estimated  location  to  see  if  these  unnecessary  recollects  could 
have  been  avoided  by  using  additional  metrics.  Overall,  we  found  32  cases  where  the  initial 
acquisition  provided  sufficient  characterization  and  the  recollect  did  not  offer  any  significant 
improvements.  Figure  23  and  Figure  24  show  two  examples  (anomalies  WK-267  and  WK-169) 
of  such  cases. 


For  several  of  these  cases,  the  initial  acquisition  was  very  close  to  the  30  cm  offset  threshold 
used  to  make  a  recollect  decision  (e.g.,  32  cm  for  WK-267  and  26  cm  for  WK-169).  For  cases 
like  these  where  the  anomaly  is  close  to  the  offset  threshold,  including  a  data/model  fit  metric 
into  the  decision  could  boost  the  operator’s  confidence.  This  information  is  currently  provided 
to  the  operator  and  it  could  be  factored  into  the  decision.  For  example,  in  both  the  WK-267  and 
WK-169  initial  encounters,  the  data/model  fit  was  very  high  (>98%).  If  the  initial  acquisition  is 
within,  say,  25  -  40  cm  of  the  anomaly,  perhaps  a  fit  of  98%  or  better  could  lead  to  a  no  recollect 
decision. 
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Figure  23.  Polarizabilities  recovered  from  the  initial  acquisition  data  (left)  and  the  recollect  data  (right). 
These  polarizabilities  correspond  to  anomaly  WK-267,  a  medium  ISO  seed.  Here  the  recollect  was 
unnecessary. 
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Figure  24.  Polarizabilities  recovered  from  the  initial  acquisition  data  (left)  and  the  recollect  data  (right). 
These  polarizabilities  correspond  to  anomaly  WK-169,  a  piece  of  frag.  Here  the  recollect  was  unnecessary. 

While  the  fit  could  be  a  good  supportive  metric  in  these  cases,  it  is  not  always  reliable. 
Consider,  for  example,  anomaly  WK-880,  a  fuze  clip  (Figure  25).  Both  the  initial  encounter  (37 
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cm  offset)  and  the  recollect  (9  cm  offset)  provided  very  good  fits  (>98%  for  both);  however,  it  is 
clear  that  the  third  polarizability  (green  line  in  Figure  25)  is  not  well  constrained  in  the  initial 
encounter  data  (although  it  could  be  argued  that  these  initial  encounter  features  are  sufficient  for 
making  a  classification  decision).  In  this  case,  we  could  add  a  model  noise  metric  (i.e., 
polarizability  noise)  to  the  decision  flow  to  determine  whether  a  recollect  is  necessary.  In  this 
case,  the  initial  set  of  polarizabilities  does  not  pass  the  model  noise  metric,  leading  to  a  recollect 
decision. 
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Figure  25.  Polarizabilities  recovered  from  the  initial  acquisition  data  (left)  and  the  recollect  data  (right). 
These  polarizabilities  correspond  to  anomaly  WK-880,  a  fuze  clip.  The  initial  data  do  not  fully  constrain  the 
third  polarizability  (green  line)  and  indicate  that  the  recollect  was  beneficial. 

Model  fit  provides  a  good  indication  that  the  target  is  well  represented  by  the  model  parameters. 
For  example  cases  where  there  are  interfering  sources,  such  as  ground  response  or  other  nearby 
targets,  will  produce  a  reduced  fit  value.  Obtaining  a  high  model  fit  (i.e.,  >98%),  however,  does 
not  necessarily  indicate  that  the  model  parameters  are  well  constrained.  The  model  noise  metric 
is  a  measure  of  the  polarizability  stability  and  is  an  indication  of  whether  or  not  the  data  provide 
good  constraint  on  the  model  parameters.  For  example,  the  initial  features  shown  in  Figure  20 
and  Figure  25  indicate  that  the  third  polarizability  (green  line)  is  unstable  and  is,  therefore,  not 
well  constrained.  Taken  together,  these  two  metrics  could  be  a  robust  way  to  assess  whether  the 
initial  acquisition  data  provide  accurate  classification  features. 

One  possible  solution  for  reducing  unnecessary  recollects  is  to  include  these  additional  metrics 
into  the  decision  flow.  This  information  is  currently  available  to  the  operator;  however,  it  could 
be  possible  to  make  this  flow  automated  to  provide  a  recollect/no  recollect  decision  for  the 
operator.  Figure  26  presents  a  possible  flow  for  including  additional  metrics.  While  reducing 
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unnecessary  recollects  is  not  critical  to  improving  the  outcome  of  the  in-field  QC  process,  it 
could  improve  the  efficiency  of  the  process. 


Figure  26.  Possible  decision  flow  for  in-field  QC.  This  decision  flow  incorporates  the  additional  fit  and  model 
noise  metrics  along  with  the  primary  target  location  metric  to  support  the  decision  to  recollect  or  not.  The 
objective  of  this  type  of  decision  flow  would  be  to  reduce  unnecessary  recollects  that  may  occur  within  some 
margin  of  error  of  the  30  cm  offset  threshold  (e.g.,  25  -  40  cm).  All  metrics  including  the  offset  threshold 
could  be  adjusted  for  site  specific  requirements  like  smallest  TOI,  target  density,  etc.  to  reflect  different 
tolerances  for  data  recollects. 

Out  of  the  32  cases  where  the  recollect  did  not  improve  target  features,  we  found  11  cases  that 
could  have  potentially  been  avoided  by  incorporating  the  basic  fit  and  model  noise  metrics  into 
the  decision  flow.  In  our  retrospective  analysis  we  set  the  fit  metric  threshold  to  a  conservative 
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value  (>98%)  such  that  these  additional  metrics  could  be  implemented  without  creating  missed 
recollect  opportunities.  Setting  a  high  value  for  the  fit  metric  was  an  effective  way  to  avoid 
missed  recollect  opportunities  as  most  of  the  “necessary”  recollect  cases  that  we  analyzed  failed 
the  initial  fit  metric  >98%. 

7.5  Objective  3.5:  Production  Rate 

For  the  production  rate  analysis,  we  wanted  to  establish  representative  production  rates  for 
surveys  performed  with  and  without  the  in-field  QC.  As  described  in  section  3.5,  we  did  not 
want  factors  specific  to  the  WMA  survey,  such  as  equipment  down  time,  equipment  transport 
time,  etc.  to  influence  these  rates.  Therefore,  we  relied  on  the  time  stamp  information  from 
groups  of  consecutively  recorded  data  files  that  were  created  during  periods  of  uninterrupted 
survey  time.  Our  time  stamp  analysis  indicated  that  with  the  in-field  QC  process,  the  field  team 
could  achieve  an  ideal  baseline  production  rate  of  23  unique  anomalies  per  hour.  Without  the  in¬ 
field  QC  process  in  place,  the  field  team  could  achieve  an  ideal  baseline  production  rate  of  27 
unique  anomalies  per  hour. 

When  these  production  rates  were  adjusted  based  on  the  number  of  necessary  recollects  for  the 
survey  (see  section  3.5.2  for  description  of  baseline  and  true  production  rate  calculations),  the 
true  production  rate  for  the  in-field  QC  approach  remained  at  23  anomalies/hr  while  the  standard 
approach  true  production  rate  dropped  to  26  anomalies/hr. 

Using  this  analysis,  it  appears  that  performing  a  survey  without  in-field  decision-based  recollects 
may  have  been  slightly  more  efficient  than  applying  the  in-field  QC  approach;  however,  a 
number  of  additional  factors  should  be  considered.  First,  a  large  number  of  recollects  (about 
66%)  appeared  to  be  a  result  of  ground  response  (i.e.,  no  contacts),  for  which  there  is  currently 
no  in-field  mitigation  process.  This  large  number  of  unnecessary  recollects  lowered  the  average 
baseline  production  rate  for  the  in-field  QC  survey.  Second,  the  number  of  necessary  recollects 
(those  that  led  to  improved  target  features)  was  relatively  small  at  46.  This  number  did  not 
significantly  lower  the  adjusted  rate  for  the  standard  approach  (it  dropped  from  27/hr  to  26/hr). 
Therefore,  we  might  expect  that  sites  without  difficult  geology  or  sites  that  contain  higher 
anomaly  densities  could  potentially  alter  these  production  rates  significantly. 

Another  way  to  assess  the  impact  of  the  in-field  QC  method  on  production  rate  is  to  view  it 
specifically  in  the  context  of  the  WMA  survey.  The  actual  production  rates  achieved  during  the 
survey  were  lower  than  the  23/hr  rate  would  indicate.  This  lower  rate  was  due  primarily  to  the 
difficult  terrain,  which  slowed  transport  of  the  sensor  across  larger  areas  of  the  site  (see  Van  et 
al.,  2015).  Actual  production  rates  achieved  were  in  the  range  of  65  -  75  anomalies  per  day. 
Based  on  the  data  time  stamp  information,  we  estimated  that  the  average  time  required  to 
perform  a  recollect  was  about  105  seconds.  Viewed  in  this  context,  the  231  recollects  probably 
added  about  6-7  hours  of  additional  survey  time.  Going  back  and  reacquiring  46  anomalies  (the 
necessary  recollects)  would  have  probably  taken  1-2  days  of  additional  survey  time  given  that 
these  anomalies  were  spread  across  different  AOI’s  within  the  site.  Thus,  it  is  likely  that  even 
with  the  recollects  associated  with  ground  response;  the  in-field  QC  enabled  a  slightly  more 
efficient  survey  for  this  particular  site. 
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7.6  Qualitative  Objectives 

We  have  discussed  ease-of-use  with  a  number  of  MetalMapper  system  operators  who  have 
performed  surveys  with  the  in-field  QC  software.  Overall,  feedback  has  been  positive.  One  of 
the  most  common  reports  is  that  having  an  inversion-based  location  that  can  be  used  to  position 
the  sensor  provides  the  operator  with  additional  confidence  in  the  quality  of  the  data.  Another 
positive  response  has  been  utility  of  the  software  for  performing  IVS  activities.  Having  the  in¬ 
field  output  for  the  polarizabilities  and  estimated  location  of  each  IVS  item  provides  immediate 
and  quantitative  feedback  about  the  functionality  of  the  system  hardware. 

8.  COST  ASSESSMENT 

The  MetalMapper  survey  costs  can  be  referenced  in  the  MR-201104  final  report  (Van  et  al., 
2015).  Total  costs  applying  the  in-field  QC  approach  during  the  survey  were  $76,100  for  cued 
acquisition  of  1032  unique  anomalies.  This  figure  produces  an  average  rate  of  $74/anomaly. 
This  rate  can  be  adjusted  for  the  standard  method  (i.e.,  no  in-field  recollects)  using  the 
aforementioned  true  production  rates  for  both  methods.  An  additional  cost  component  is  the 
time  required  for  set  up  and  installation  of  the  in-field  QC  software. 


Table  4.  Cost  Model  for  In-Field  Quality  Control  for  Cued  EMI  Surveys 


Data  to  be  Tracked 

Survey  costs  (in-field  QC) 

$74/anomaly 

•  Assumes  1032  unique  anomalies  surveyed  at  a  rate 
of  23  unique  anomalies  per  hour 

•  Includes  153  recollects  associated  with  ground 
response 

Survey  costs  (standard) 

$65/anomaly 

•  Assumes  1032  unique  anomalies  surveyed  at  a  rate 
of  26  unique  anomalies  per  hour 

•  Includes  46  reacquisitions 

T  raining/installation 

$2400 

•  Approximately  Vi  day  of  survey  time 

8.1  Cost  Drivers 


Production  rate  analysis  indicates  that  the  cost  per  anomaly  may  have  been  lower  if  the  survey 
had  been  applied  without  recollects.  This  is  due  to  the  fact  that  a  large  number  of  recollects 
(153)  were  associated  with  ground  response,  which  produced  inaccurate  source  locations. 
Additionally,  there  were  relatively  few  recollects  (46  out  of  231  total)  that  provided  significant 
improvements  in  target  features.  These  cost  drivers  reflect  the  challenges  that  were  specific  to 
the  WMA  site. 

The  true  cost  comparison  of  the  two  methods  for  the  WMA  site  may  be  somewhat  different  than 
what  is  reflected  by  the  cost  per  anomaly  in  Table  5  if  the  specific  conditions  of  the  site  are 
considered.  For  example,  it  is  unlikely  that  reacquisition  of  46  targets  could  actually  be 
accomplished  in  ~2  hours.  In  reality,  this  would  likely  have  required  1-2  additional  survey 
days  given  that  these  anomalies  covered  3  different  AOI’s  and  transport  between  these  areas  was 


White  River  Technologies 


37 


January  2016 


EST CP -MR-201264  Real  Time  Quality  Control  Methods  for  Cued  EMI  Data  Collection 


time  consuming.  We  expect  that  the  true  costs  for  each  method  at  this  particular  site  were 
comparable,  if  not  slightly  lower  for  the  in-field  QC  method. 

9.  IMPLEMENTATION  ISSUES 

The  greatest  challenge  to  effective  implementation  of  the  in-field  QC  process  at  the  WMA  site 
was  the  presence  of  significant  magnetic  geology  throughout  the  AOI’s.  Our  retrospective 
analysis  of  the  survey  data  indicated  that  a  significant  portion  of  recollects  based  on  the  in-field 
decision  were  a  result  of  the  soil  response  (153  out  of  231).  After  analyzing  the  recollect  data, 
we  have  developed  a  few  recommendations  that  could  improve  the  effectiveness  and  efficiency 
of  the  in-field  QC  process  at  other  challenging  sites: 

•  One  approach  to  enable  in-field  mitigation  of  magnetic  geology  could  be  to  enhance  the 
background  selection  and  removal  process  of  the  in-field  software.  For  example, 
multiple  background  locations  based  on  DGM  analysis  of  magnetic  geology  in  the  survey 
area  could  be  included  in  the  background  selection  process  for  the  in-field  software. 
Background  files  collected  at  these  locations  could  be  loaded  into  the  module  and 
automatically  selected  based  on  location  of  the  sensor.  Proximity  to  different  background 
locations  in  the  site  would  drive  the  selection  of  the  optimal  background  file  to  apply 
before  inverting  each  subsequent  data  file. 

•  In  many  cases,  however,  there  will  not  be  an  ideal  background  file.  This  was  evident  in 
the  post-survey  processing  when  it  became  apparent  that  none  of  the  background  files 
collected  on  a  given  day  was  adequate  for  removing  the  ground  response  for  a  number  of 
anomaly  encounters.  In  cases  such  as  these,  more  sophisticated  inversion  strategies  are 
required.  For  example,  replicating  the  ground  response  by  constraining  the  location  of  a 
deep  dipole  source  during  inversion  has  been  effective  for  isolating  the  response  from 
targets  buried  in  magnetic  soil  (see  e.g.,  Pasion  et  al.,  2012).  It  could  be  an  effective 
approach  to  include  a  2-source  solver  in  the  in-field  model  that  would  allow  one  source  to 
be  constrained  to  represent  the  ground  response.  Decay  fit  metrics  could  be  applied  to 
determine  if  the  constrained  source  does  indeed  produce  ground-like  features  that 
indicate  the  presence  of  significant  magnetic  geology. 

•  While  the  WMA  survey  did  not  appear  to  produce  many  cases  where  multiple  objects 
influenced  the  in-field  QC  decision,  there  are  sites  where  this  could  be  an  issue.  In  some 
cases,  implementing  a  multi-source  solver  could  be  effective  for  isolating  the  ground 
response  (see  above)  as  well  as  for  isolating  responses  from  multiple,  closely  spaced 
targets. 

•  Finally,  using  quality  metrics  in  addition  to  the  estimated  source  location  could  prove 
useful  for  eliminating  unnecessary  recollects.  In  cases  where  the  target  is  isolated  from 
any  interfering  sources  (e.g.,  ground  response,  other  nearby  targets,  low  SNR,  etc.),  the 
30  cm  offset  threshold  may  be  overly  stringent.  While  eliminating  unnecessary  recollects 
may  not  be  critical  to  improving  the  effectiveness  of  the  process,  it  could  improve  the 
efficiency  of  the  process,  leading  to  better  overall  cost  performance.  Incorporating  other 
metrics  into  the  decision  flow  (see  Figure  26  in  section  7.4  for  example)  could  eliminate 
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recollects  in  some  of  these  cases  where  the  initial  acquisition  provides  adequate 
characterization  of  the  target. 

While  the  WMA  survey  elucidated  several  challenges  for  the  in-field  QC  process,  we  believe 
that  overall  the  process  proved  beneficial  for  the  survey.  There  were  46  cases  where  the 
recollect  provided  significant  improvements  in  the  characterization  of  the  target  and  most  likely 
improved  the  final  classification  result.  Given  the  challenges  of  transporting  and  moving 
equipment  around  the  different  AOI’s,  these  46  anomalies  would  likely  have  added  an  additional 
1-2  days  of  surveying  if  they  had  not  be  recollected  based  on  the  in-field  decision  (for 
comparison,  we  estimate  the  231  total  recollects  for  the  survey  added  6-7  hours  of  actual  survey 
time  over  the  course  of  field  activities).  Additionally,  there  are  other  benefits  to  providing  the 
field  team  with  immediate  inversion-based  metrics  including  the  immediate  feedback  about 
instrument  functionality  during  IVS  activities. 

It  should  also  be  noted  that  during  most  of  the  survey,  one  of  the  comer  receivers  on  the 
MetalMapper  was  malfunctioning.  It  is  unclear  to  what  extent  this  faulty  receiver  influenced  the 
in-field  decisions;  however,  it  is  possible  that  it  had  some  effect  on  the  accuracy  of  in-field  target 
location  estimates. 

It  is  likely  that  a  large  number  of  sites  could  benefit  from  implementing  an  in-field  QC  process 
for  cued  surveys.  Applying  some  of  the  aforementioned  improvements  could  make  the  process 
more  effective  for  particularly  challenging  sites,  such  as  those  containing  magnetic  geology  or 
high  anomaly  densities. 
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Appendix  A  -  Interim  Proveout  Report 


The  following  pages  include  an  interim  report  submitted  after  the  initial  proveout  phase  of  the 
project.  This  report  provides  a  detailed  description  of  the  in-field  QC  hardware  and  procedures 
and  demonstrates  the  feasibility  of  including  this  process  during  a  cued  survey. 
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1.  In-Field  Quality  Control 

Currently,  standard  protocol  for  collecting  advanced  EMI  sensor  data  includes  static  data 
acquisition  over  anomalies  identified  (cued)  by  prior  dynamic  or  Digital  Geophysical  Mapping 
(DGM)  surveys.  These  data  are  typically  analyzed  by  off-site  geophysicists  upon  completion  of 
the  cued  survey.  Often,  this  analysis  results  in  a  number  of  anomalies  that  are  found  to  be 
insufficiently  characterized  by  the  initial  cued  data  collection  as  a  result  of  sub-optimal 
placement  of  the  EMI  sensor  during  the  survey.  Subsequently,  reacquisition  of  data  is  necessary 
and  additional  surveying  is  required.  Methods  that  significantly  reduce  reacquisitions  and 
improve  the  overall  quality  of  the  data  before  completion  of  the  initial  cued  survey  will 
significantly  improve  the  efficiency  of  production  operations. 

Because  cued  EMI  sensors  rely  on  multi-axis  illumination  to  provide  enough  information  to 
extract  useful  classification  features  from  the  data,  the  placement  of  the  sensor  relative  to  the 
object  under  interrogation  is  critical.  In  many  instances  when  the  cued  data  yield  poor 
classification  features,  the  reason  is  often  an  incorrectly  positioned  sensor.  Positioning  errors, 
misguided  target  picking,  or  multiple  objects  in  the  field  of  view  are  common  causes  of  incorrect 
cued  sensor  placement.  Most  often,  these  errors  are  not  discovered  until  the  off-site  geophysicist 
identifies  them  during  analysis  of  the  data.  Once  discovered,  the  analyst  will  determine  a 
rectified  set  of  coordinates  to  better  represent  the  true  location  of  the  target.  The  field  team  will 
then  reacquire  the  data  by  positioning  the  sensor  over  the  new  coordinates. 

Providing  the  field  team  with  the  capability  to  assess  data  quality  immediately  following  the 
initial  data  collection  will  significantly  improve  the  cued  survey  process.  By  removing  the  off¬ 
site  analyst  from  the  initial  quality  decision  making,  the  field  team  can  take  immediate  action  to 
reposition  and  recollect  certain  targets  if  deemed  necessary.  Immediate  recollection  based  on  in¬ 
field  quality  control  will  be  much  more  efficient  than  subsequent  reacquisition  based  on  off-site 
decision  making.  Offsite  analysis  will  still  be  important  for  identifying  complicated  target 
scenarios  (e.g.,  multi-object,  magnetic  geology,  etc.);  however,  many  of  the  reacquires  due  to 
common  errors  such  as  inaccurate  target  picking  could  be  replaced  by  in-field  decision  based 
recollects. 

1.1.  Quality  Control  Technology  Proveout 

During  the  initial  phase  of  this  project,  we  evaluated  an  in-field  quality  control  software  module 
to  provide  real-time  analysis  of  cued  EMI  data.  For  demonstration  purposes,  we  have 
specifically  designed  the  QC  module  to  analyze  MetalMapper  data.  Because  the  MetalMapper  is 
a  commercially  available  sensor  that  has  been  selected  for  a  number  of  live  site  demonstrations, 
we  believe  that  applying  real-time  quality  control  to  the  MetalMapper  data  acquisition  process  is 
highly  relevant  to  ESTCP’s  need  for  production  technologies  that  have  the  ability  to  collect 
classification-quality  survey  data.  It  should  be  noted,  however,  that  while  the  QC  module  is 
currently  optimized  for  MetalMapper  surveys,  it  can  easily  be  modified  to  analyze  any  cued  EMI 
sensor  data. 

The  QC  module  provides  three  principal  capabilities: 
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1.  Real-time  inversion  of  cued  MetalMapper  data  to  identify  quality  metrics  while  the 
MetalMapper  is  still  within  the  vicinity  of  the  cued  anomaly  (on  the  order  of  1  second); 

2.  Analysis  of  quality  metrics  to  enable  the  user  to  make  an  informed  decision  as  to  whether 
the  data  are  of  sufficient  quality  to  advance  to  the  next  anomaly  or  whether  recollection  is 
required; 

3.  Guidance  for  optimal  placement  of  the  sensor  in  the  event  that  data  recollection  is 
required. 

This  project  comprises  two  phases:  (1)  a  technology  proveout  phase,  and  (2)  a  technology 
demonstration  phase.  We  verified  the  aforementioned  capabilities  during  the  initial  proveout 
phase  of  this  project  through  a  series  of  tests  conducted  at  the  Former  Lowry  Bombing  and 
Gunnery  Range  (FLBGR)  in  Aurora,  Colorado.  These  tests  included  simulated  cued  surveys  as 
well  as  controlled  test  pit  data  collections  over  standard  test  objects. 

In  order  for  the  project  to  proceed  to  a  live  site  demonstration  of  the  in-field  QC  technology,  the 
QC  module  must  meet  the  following  requirements  to  verify  its  principal  capabilities: 

1.  Inversion  of  static  MetalMapper  data  and  display  of  dipole-fit  parameters  within  1  second 
(nominal)  of  acquisition  completion; 

2.  Display  of  sensor  position  within  10  cm  of  true  GPS  coordinates; 

3.  Display  of  estimated  dipole/target  location  within  10  cm  of  true  anomaly  location. 

This  report  summarizes  the  results  of  the  proveout  tests  and  provides  confirmation  that  the  QC 
module  meets  the  performance  requirements  for  advancing  the  project  to  a  live  site 
demonstration  phase. 

2.  In-Field  QC  Module 

The  QC  module  functions  as  a  parallel  process  to  the  cued  EMI  sensor  data  acquisition  software. 
As  such,  the  QC  module  does  not  provide  any  data  acquisition  functionality;  it  is  used  solely  for 
immediate  in-field  analysis  of  the  cued  sensor  data.  The  QC  module  uses  a  file  watch  protocol, 
which  means  that  it  loads  cued  sensor  data  files  immediately  after  they  are  created  in  the  data 
acquisition  file  directory.  In  this  sense,  the  QC  software  is  truly  modular  in  that  it  can  be  used  in 
parallel  with  any  data  acquisition  software,  provided  the  associated  data  file  formats  are 
accepted. 

Initially,  we  considered  a  design  that  would  more  closely  integrate  the  QC  software  with  the  data 
acquisition  software.  This  design  would  have  combined  the  inversion  and  classification 
Application  Programming  Interface  (API),  which  provides  the  backbone  for  the  QC  module, 
with  the  acquisition  software  to  form  a  single  process.  To  implement  this  process  for  the 
MetalMapper,  several  methods,  events,  and  properties  found  in  the  MetalMapper  data  acquisition 
software,  known  as  EM3D,  would  need  to  be  declared  “public”  in  the  C#  code.  This  would 
allow  the  API  to  gain  direct  access  to  the  acquired  data  rather  than  rely  on  the  file  watch 
protocol. 

The  intent  with  this  approach  was  that  it  would  create  a  seamless  process  where  the  EM3D 
software  would  provide  both  the  data  acquisition  and  sensor  navigation  interfaces,  the  latter  of 
which  would  receive  updates  from  the  classification  API.  This  approach  would  essentially 
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maintain  the  existing  functionalities  of  the  EM3D  software  and  give  it  an  added  feature  of 
providing  real-time  data  inversion  capabilities.  While  providing  EM3D  with  this  feature  would 
build  an  in-field  QC  capability  into  the  MetalMapper  sensor  package,  we  determined  that  this 
approach  would  limit  options  for  adapting  the  QC  capability  to  other  sensor  forms.  Additionally, 
because  EM3D  is  currently  being  transitioned  to  commercial  use,  we  felt  that  modifying  any  of 
its  features  might  impede  this  commercialization  process. 

Instead,  we  focused  on  developing  a  software  executable  that  was  completely  modular  and  could 
be  adapted  to  any  cued  sensor  data  format.  Our  initial  concerns  about  the  file  watch  approach 
not  being  robust  enough  for  production  operations  proved  to  be  unfounded.  We  performed 
extensive  tests  with  our  field  teams  using  the  QC  module  in  a  production  environment  over  a 
period  of  several  weeks.  Not  only  was  the  file  watch  process  100%  reliable  during  these 
operations,  but  it  was  completely  transparent  to  the  MetalMapper  operators,  i.e.,  the  results 
obtained  using  the  file  watch  protocol  were  effectively  instantaneous.  There  is  no  noticeable  lag 
that  occurs  between  the  data  acquisition  process  and  the  QC  module  data  inversion.  Observing 
the  reliability  of  the  file  watch  process  convinced  us  that  this  approach  was  optimal  for  enabling 
future  transition  of  the  module  to  cued  sensor  applications  that  extended  beyond  just 
MetalMapper  surveys. 


The  result  of  our  modified  design 
concept  is  a  QC  module  that 
provides  both  navigation  and  quality 
analysis  interfaces.  We  currently 
maintain  use  of  EM3D  solely  for 


With  our  initial  development  concept  shifted  towards  a  more  modular  design,  we  required  that 
the  QC  module  take  on  some  of  the  tasks  normally  assigned  to  EM3D.  Specifically,  the  QC 
module  needed  to  provide  some  navigation  functionality.  We  determined  that  this  capability 
would  be  relatively  easy  to  develop  and,  more  importantly,  that  it  would  be  a  critical  component 
of  a  modular  QC  interface.  Because 
the  ability  to  reposition  the  sensor 
head  based  on  data  inversion  results 
is  a  key  component  of  in-field  QC,  it 
was  a  logical  step  to  include  this 
capability  as  part  of  a  QC  navigation 
interface  that  functioned  separately 
from  the  data  acquisition  software. 

Additionally,  any  cued  sensor  will 
require  sensor  head  positioning 
information,  typically  in  the  form  of 
GPS  and  Inertial  Measurement  Unit 
(IMU)  data  streams.  Thus,  it  would 
be  relatively  straightforward  to  split 
these  incoming  positioning  data 
streams  between  the  data  acquisition 
software  and  the  QC  module. 


Figure  1.  Block  diagram  representing  data  flow  patterns 
between  various  hardware  (HW)  and  software  (SW) 
components. 
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data  acquisition  capabilities  that  enable  selection  of  MetalMapper  data  acquisition  parameters  as 
well  as  triggering  of  data  acquisition  events.  A  block  diagram  that  shows  the  separate 
functionalities  of  the  acquisition  and  QC  software  components  is  presented  in  Figure  1. 

In  practice,  operating  the  EM3D  acquisition  software  and  the  QC  module  in  parallel  is 
straightforward  and  does  not  require  any  shuffling  of  interfaces  to  acquire  data.  Our  initial 
concern  in  having  separate  data  acquisition  and  data  QC  processes  was  that  it  might  be  onerous 
for  the  operator  to  switch  back  and  forth  between  the  data  acquisition  interface  for  triggering  an 
acquisition  event  and  the  QC  interface  for  viewing  results;  however,  in  reality  this  process 
requires  very  few  steps.  Once  the  initial  data  acquisition  parameters  are  set  for  the  MetalMapper 
using  EM3D,  the  QC  module  can  be  used  to  provide  the  primary  navigation  and  quality  analysis 
interfaces.  Only  a  small  trigger  button  from  the  EM3D  interface  must  remain  accessible  for 
commencing  a  data  acquisition  event.  Figure  2  shows  an  operator  display  that  demonstrates  the 
typical  setup  for  enabling  both  QC  and  data  triggering  functions. 
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Figure  2.  Typical  operator  display  setup  showing  QC  and  acquisition  interfaces.  The  QC  Module  provides 
the  main  navigation  and  analysis  interfaces,  while  EM3D  provides  the  acquisition  capability  with  a  simple 
acquire  button. 
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As  mentioned  previously,  the  QC  module  is  built  around  our  C++  inversion  and  classification 
API  that  provides  access  to  the  inversion  algorithms,  which  can  be  used  to  rapidly  estimate  a 
variety  of  model  parameters  associated  with  each  anomaly.  These  parameters  include  principal 
target  polarizabilities  and  estimated  target  location.  When  the  QC  module  executable  performs  a 
file  watch  on  the  cued  data  directory,  any  acquisition  event  triggered  by  the  EM3D  interface  will 
create  a  new  file  in  this  directory.  The  QC  module  will  immediately  load  this  file  and  invert  the 
data  contained  within  it.  The  output  of  these  inversions  is  subsequently  transferred  to  the 
operator  navigation  and  QC  interfaces  that  display  the  various  recovered  dipole-model 
parameters  (e.g.,  polarizabilities,  location,  depth,  etc.)  as  well  as  the  raw  data  channel  output. 

The  versatility  of  the  API  highlights  the  modularity  of  the  QC  executable.  The  API  can  be 
adapted  to  a  variety  of  sensors.  Before  the  API  can  work  with  new  sensor  data,  information 
about  the  sensor  configuration  is  required.  The  sensor  configuration  tells  the  API  how  the  data 
are  collected;  it  consists  of  specifications  such  as  the  geometry  of  the  sensors  with  respect  to  the 
GPS  sensor  location,  the  types  of  sensors  used,  the  geometry  of  transmitter  and  receiver  coils,  the 
number  of  time  gates  used,  etc.  This  information  enables  an  initialization  of  the  API  once  for 
any  given  cued  survey  sensor.  After  the  operator  sets  the  sensor  configuration,  the  calling 
application  will  initialize  the  inversion  configuration,  which  determines  the  inversion  method  (a 
specific  forward  model),  the  parameter  scaling,  parameter  bounds,  solver  stopping  criteria,  and 
additional  options  that  control  the  inversion. 

2.1.  QC  Module  Setup  and  Features 

Step  1.  Building  a  Reference  Library:  The  first  step  in  implementing  the  in-field  QC  process  is 
to  establish  a  target  library.  The  QC  module  facilitates  this  by  allowing  the  operator  to  create 
reference  polarizability  libraries  for  standard  objects  as  well  as  for  objects  that  may  be 
indigenous  to  the  site.  The  operator  starts  by  opening  the  QC  module  flow  that  enables  the 
creation  of  a  library  file.  Within  the  flow,  the  operator  can  select  a  background  data  file  to  use 
for  processing  of  subsequent  acquisition  files.  The  operator  also  indicates  the  location  of  the 
data  file  directory  where  EM3D  deposits  the  acquisition  files  (Figure  3). 


White  River  Technologies 


45 


January  2016 


ESTCP -MR-201264  Real  Time  Quality  Control  Methods  for  Cued  EMI  Data  Collection 


MetalMapper  Background  File 


Data  File  Directory 


Figure  3.  Library  Create  Flow.  The  operator  selects  a  background  data  file  as  well  as  the  location  of  the 
incoming  data  file  directory. 

Once  the  flow  functions  are  established,  the  operator  selects  the  “RUN”  button  to  start  the  flow. 
This  will  activate  the  file  watch  mode  and  the  QC  module  will  wait  for  EM3D  to  generate  a 
MetalMapper  data  file  in  the  directory.  The  operator  will  then  place  a  library  object  under  the 
MetalMapper  sensor  head  and  select  the  “Acquire”  button  in  the  EM3D  acquisition  interface. 
Once  the  MetalMapper  data  file  is  created,  the  QC  module  will  invert  the  data  and  display  the 
quality  analysis  interface  for  the  library  object  (Figure  4).  If  the  recovered  polarizabilities  are 
acceptable  for  library  cataloguing,  the  operator  will  check  the  ground  truth  box  and  select  a 
target  type  corresponding  to  the  library  object.  This  process  can  then  be  repeated  for  a  number 
of  library  objects.  Once  the  library  is  fully  populated  with  the  desired  object  polarizabilities,  the 
operator  can  save  the  library  file  for  future  reference. 
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Figure  4.  Quality  analysis  interface  showing  library  object  features.  If  the  target  polarizabilities  are 
acceptable,  the  operator  can  add  them  to  the  library  by  checking  the  ground  truth  box  and  selecting  the 
target  type. 

For  best  library  results,  the  library  data  should  be  collected  in  a  clean  area  such  as  a  test  pit  or 
calibration  zone  so  that  the  data  will  reflect  only  the  target  response.  Additionally  each  library 
object  should  be  placed  directly  under  the  center  of  the  sensor  head  to  ensure  full 
characterization  of  the  target  features  (Figure  5).  Background  files  should  also  be  collected  in 
the  same  area  without  any  target  present  to  provide  sufficient  background  response 
characterization. 
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Figure  5.  Library  data  collection.  Library  objects  are  placed  in  a  clean  area  such  as  test  pit  (left).  Once 
centered,  the  sensor  head  is  lowered  over  the  object  for  data  collection  (right). 

Step  2.  Loading  Target  Picks:  Once  the  library  file  is  created,  the  operator  must  load  a  target 
pick  file  into  the  QC  module  before  commencing  cued  survey  operations.  This  step  provides  the 
QC  module  with  the  location  of  each  target  so  that  the  coordinates  of  these  picks  can  be 
displayed  in  the  navigation  interface.  The  target  file  features  a  basic  .xyz  file  format  that  assigns 
an  easting,  northing,  and  target  ID  number  to  each  pick.  The  operator  can  load  the  target  file  by 
opening  the  QC  module  flow  that  creates  an  SRD  file  (Figure  6).  This  flow  enables  selection  of 
a  specific  target  file  that  corresponds  to  the  anticipated  working  area  of  the  site.  Once  the  correct 
target  file  is  selected,  the  operator  can  select  the  “RUN”  button  to  load  the  target  pick 
coordinates  into  the  QC  module.  Running  this  flow  will  create  an  SRD  file  that  corresponds  to 
the  target  picks  in  the  selected  target  file.  The  SRD  file  will  be  used  in  subsequent  cued  survey 
operations  to  record  model  information  (such  as  recovered  polarizabilities  and  estimated 
location)  for  the  data  files  corresponding  to  each  target  pick. 
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Target  Pick  File  Selection 


Figure  6.  Create  SRD  Flow.  This  flow  allows  the  operator  to  load  target  files  into  the  QC  module.  The  target 
pick  coordinates  will  then  be  displayed  in  the  navigation  display  during  subsequent  cued  survey  operations. 

Step  3.  Opening  the  Navigation  and  QC  Interfaces:  After  loading  the  target  pick  coordinates,  the 
operator  can  view  the  navigation  and  QC  interfaces  by  opening  the  Navigation  and  QC  Flow 
(Figure  7).  This  flow  enables  selection  of  the  SRD  file  (created  in  Step  2)  that  corresponds  to 
the  target  picks  located  in  the  anticipated  working  area.  Within  this  flow,  the  operator  can  also 
select  a  reference  library  file  (created  in  Step  1),  a  MetalMapper  background  file,  and  the 
location  of  the  data  directory  where  EM3D  will  create  new  data  files  during  the  cued  survey. 
Once  these  flow  functions  are  correctly  assigned,  the  operator  can  select  the  “RUN”  button  to 
open  the  navigation  and  QC  interfaces. 
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Load  SRD  file  Select  Reference  Library  Select  Background  File 


Select  Data  File  Directory 

Figure  7.  Navigation  and  QC  Flow.  The  operator  assigns  an  SRD  file,  a  library  file,  a  background  file,  and 
the  data  directory  in  these  flow  functions.  Running  the  flow  opens  the  navigation  and  QC  interfaces. 

Running  the  Navigation  and  QC  Flow  automatically  opens  the  navigation  interface,  which 
displays  the  coordinates  of  each  pick  in  the  target  file  as  well  as  the  location  of  the  MetalMapper 
sensor  head  (Figure  8).  The  target  picks  are  displayed  using  a  vehicle -referenced  coordinate 
system.  Thus,  the  MetalMapper  location  is  always  fixed  in  the  center  of  the  display  with  the 
vehicle  heading  fixed  towards  the  top  of  the  screen.  The  targets  are  queued  such  that  the  next 
target  for  surveying  is  shown  in  red  and  the  remaining  picks  are  shown  in  blue. 
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Next  Target  MetalMapper  Sensor 


Figure  8.  Navigation  interface.  Target  pick  (red  and  blue  dots)  locations  are  shown  relative  to  the 
MetalMapper  sensor  head. 

Step  4.  Starting  a  Cued  Survey:  With  the  available  target  picks  displayed  in  the  navigation 
interface,  the  operator  may  use  the  interface  to  select  the  next  target  for  surveying.  As  the 
operator  drives  towards  the  next  target,  the  display  screen  will  rotate  to  reflect  the  changing 
heading.  As  the  sensor  head  approaches  the  target,  the  display  auto-zooms  to  facilitate 
positioning  of  the  sensor  head  directly  over  the  target.  Once  the  sensor  is  centered  over  the 
target,  the  operator  will  select  the  EM3D  acquisition  button  to  acquire  a  data  file.  When  the 
acquisition  is  complete,  EM3D  will  generate  a  .tern  data  file  in  the  directory.  The  QC  module 
then  uses  the  API  to  invert  the  data  stored  in  this  file.  The  QC  interface  will  automatically  open 
to  display  the  results  when  the  inversion  is  complete.  These  results  include  the  recovered 
polarizabilities,  the  estimated  target  location,  and  a  correlation  coefficient  that  indicates  how  well 
the  model  results  match  the  data  (Figure  9).  The  polarizabilities  are  plotted  against  the  best 
library  match  to  indicate  the  likelihood  that  the  target  is  one  of  the  items  catalogued  in  the 
library.  The  estimated  target  location  will  also  be  displayed  in  the  navigation  interface. 
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Figure  9.  Navigation  interface  (left)  and  QC  interface  (right)  displaying  the  data  inversion  results 
corresponding  to  the  current  target  pick  (number  1).  The  original  pick  coordinates  are  represented  by  the 
red  dot  and  the  estimated  target  coordinates  are  represented  by  the  pink  dot.  The  recovered  polarizabilities 
(colored  lines)  are  plotted  in  the  QC  interface  along  with  the  best  library  match  (grey  lines).  In  this  case,  the 
recovered  polarizabilities  show  an  exact  match  with  the  library  polarizabilities  for  this  item  so  it  is  difficult  to 
see  the  library  features  in  the  plot. 

Based  on  the  inversion  results,  the  operator  can  assess  the  quality  of  the  data  and  decide  whether 
the  data  are  of  sufficient  quality  or  whether  a  recollection  is  necessary.  If  a  recollection  is 
required,  the  operator  selects  the  “Fail”  button  on  the  QC  interface.  This  action  will  update  the 
current  target  pick  coordinates  based  on  the  estimated  target  location  (the  original  coordinates 
are  still  stored  in  the  SRD  file).  The  operator  can  then  navigate  using  this  new  set  of  coordinates 
and  position  the  sensor  head  over  the  estimated  target  location  to  recollect  the  data.  Once  the 
data  are  determined  to  be  of  sufficient  quality,  the  operator  can  advance  to  the  next  target  in  the 
queue  and  repeat  the  cued  acquisition  sequence. 

Step  5.  Analyzing  the  SRD  Data:  After  the  cued  survey  is  completed  for  all  targets  in  the  pick 
file,  the  survey  quality  statistics  (e.g.,  percentage  of  targets  requiring  recollects,  model 
correlation  distribution,  etc.)  may  be  compiled  by  analyzing  the  data  stored  in  the  SRD  file.  This 
analysis  is  typically  performed  offsite  since  it  does  not  affect  any  of  the  in-field  QC  processes. 
The  SRD  file  contains  all  of  the  metadata  for  the  survey  as  well  as  the  data  inversion  results 
corresponding  to  each  data  file  collected  during  the  survey. 

The  SRD  file  can  be  opened  at  any  point  after  the  survey  is  completed  using  the  QC  module. 
The  results  for  each  acquisition  can  be  viewed  using  a  post-survey  analysis  interface. 
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Additionally,  the  QC  module  provides  a  function  to  export  the  data  and  results  to  MATLAB  for 
more  detailed  analysis  (Figure  10). 
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Figure  10.  SRD  data  analysis.  The  QC  module  provides  a  post-survey  analysis  interface  (top)  for  opening 
SRD  files,  as  well  as  a  function  that  exports  the  SRD  data  to  MATLAB  (bottom). 


3.  Data  Quality  Metrics 

Effective  data  quality  metrics  are  critical  to  ensuring  successful  in-field  quality  decisions.  As 
part  of  the  initial  proveout  phase  of  this  project,  we  undertook  an  extensive  data  analysis  study  to 
identify  the  most  practical  and  effective  set  of  quality  metrics  to  apply  to  our  in-field  QC 
software  module. 
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Over  the  course  of  numerous  live  site  demonstrations  of  cued  EMI  sensors,  it  has  been  well 
established  that  sensor  position  relative  to  the  target  is  important  for  ensuring  the  data  contain 
good  classification  information.  Because  most  cued  sensors  rely  on  multi-axis  magnetic  field 
illumination  of  the  target  to  enable  effective  characterization,  these  sensors  must  be  positioned 
such  that  they  achieve  excitation  of  all  three  principal  axes  of  the  target.  For  the  MetalMapper, 
optimal  sensor  positioning  requires  that  the  MetalMapper  sensor  head  is  centered  directly  above 
the  target.  Poor  sensor  positioning  can  often  result  in  poor  classification  results  (Figure  1 1). 
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Figure  11.  MetalMapper  data  collected  during  the  Pole  Mountain,  WY  live  site  demonstration.  The  initial 
acquisition  was  conducted  with  poor  sensor  placement  relative  to  the  target  (top).  Consequently,  the  data 
produced  poor  classification  features.  Data  were  subsequently  reacquired  using  better  sensor  placement 
(bottom),  which  yielded  good  classification  features  for  the  target. 

Because  classification  quality  correlates  strongly  to  sensor  positioning  quality,  the  estimated 
target  location  is  a  fairly  robust  quality  metric;  however,  there  are  instances  where  somewhat 
misguided  sensor  placement  can  still  produce  data  that  yield  good  classification  features.  The 
key  objective  of  our  data  analysis  study  was  to  identify  additional  quality  metrics  that  could 
further  guide  quality  decisions  that  were  initially  based  on  sensor  positioning  alone.  These 
metrics  could  be  used  to  identify  cases  where  relatively  large  offsets  between  the  target  and  the 
sensor  still  resulted  in  good  classification.  In  these  instances,  recollection  would  be  unnecessary. 
Thus,  these  additional  metrics  could  further  improve  survey  efficiency  by  reducing  the  number 
of  data  recollects. 

3.1.  Test  Pit  Data  Compilation 

To  assess  the  merits  of  potential  quality  metrics,  we  compiled  a  set  of  controlled  data.  We 
collected  over  600  unique  MetalMapper  measurements  at  our  test  pit  at  the  FFBGR  site.  This 
data  set  provided  a  statistically  large  sample  of  test  cases  for  identifying  data  quality  metrics. 
For  each  test  case,  we  placed  a  standard  UXO  test  object  (e.g.,  37mm  projectile,  60  mm  mortar, 
small  ISO40,  etc.)  on  a  grid  surface  within  the  test  pit  (Figure  12). 


White  River  Technologies 


54 


January  2016 


ESTCP -MR-201264  Real  Time  Quality  Control  Methods  for  Cued  EMI  Data  Collection 


\ 


Figure  12.  Test  pit  data  collection.  Objects  were  placed  on  a  test  grid  beneath  the  MetalMapper  sensor  head 
(left).  Test  cases  comprised  mostly  single  standard  test  objects  placed  at  various  offsets  from  the  grid  center 
(top  right);  however,  some  cases  included  clutter  samples  to  increase  noise  (bottom  right). 

Because  we  were  interested  in  identifying  data  or  model  parameters  that  would  indicate  the 
classification  quality  of  the  data,  we  needed  to  ensure  that  our  data  set  comprised  samples  that 
produced  both  high  quality  and  low  quality  classification  features.  To  ensure  a  diverse  set  of 
samples,  we  varied  a  number  of  test  parameters  including  target  depth,  orientation,  and  lateral 
offset.  Over  the  course  of  the  data  collection,  we  moved  each  test  object  to  a  number  of 
positions  on  the  grid,  ranging  from  centered  directly  beneath  the  MetalMapper  (i.e.,  grid  center) 
to  lateral  offsets  of  up  to  70  cm  from  the  center  of  the  MetalMapper.  We  varied  the  depth  and 
orientation  for  each  test  case,  and  we  also  included  clutter  items  in  some  measurements  to 
increase  noise  levels.  These  variations  provided  a  diverse  data  set  that  included  both  low  and 
high  Signal-to-Noise  Ratio  (SNR)  samples. 

It  should  be  noted  that  while  we  did  conduct  test  cases  that  included  multiple  objects,  the  focus 
of  our  testing  was  acquiring  data  on  which  we  would  apply  single  object  inversions.  Because  the 
QC  module  currently  implements  a  single  object  inversion,  we  wanted  to  establish  the  limits  of 
this  method  even  in  cases  where  multiple  objects  were  within  the  sensor  field  of  view.  We  are 
currently  compiling  a  more  extensive  set  of  multiple  object  cases.  We  will  eventually  use  these 
data  to  establish  metrics  that  will  indicate  the  reliability  of  single  object  inversions  and  identify 
the  necessity  for  multiple  object  inversions  within  the  QC  module. 
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3.2.  Quality  Metrics  Analysis 

After  compiling  an  extensive  set  of  test  pit  data,  we  performed  an  in-depth  analysis  to  identify 
data  and  model  parameters  that  could  be  used  to  indicate  the  classification  quality  of  the  data. 
To  begin  this  process,  one  of  our  classification  analysts  conducted  a  visual  QC  analysis  of  the 
inversion  results  from  each  test  case.  He  evaluated  the  recovered  polarizabilities  from  each 
sample  and  determined  whether  these  produced  good  or  insufficient  classification  features. 
Specifically,  each  case  was  selected  for  one  of  three  categories:  1)  cases  that  yielded  good 
classification  features;  2)  those  that  yielded  a  good  primary  polarizability,  but  poor  secondary  or 
tertiary  polarizabilities;  and  3)  those  that  did  not  produce  any  good  classification  features. 
Examples  from  these  categories  are  shown  in  Figure  13. 


kw  #1/1=St  1/1  (pass) 


kw  #1/1=SI:  1/1  (pass) 


kw  #1/1=SI:  1/1  <(ai) 


Figure  13.  Examples  of  test  pit  data  that  produced  poor  classification  features:  (left)  one  bad  polarizability; 
(center)  two  bad  polarizabilities;  (right)  no  good  polarizabilities. 

We  were  particularly  interested  in  identifying  cases  where  the  data  produced  good  classification 
features,  but  the  target  offset  was  large.  A  typical  rule  of  thumb  for  quantifying  large  offsets  is 
to  select  cases  where  the  target  lateral  offset  is  greater  than  30  cm  from  the  center  of  the  sensor 
head.  While  30  cm  is  fairly  conservative  (i.e.,  it  is  possible  to  achieve  good  classification  for 
targets  well  outside  this  range),  it  provides  a  practical  threshold  since  most  production  surveys 
will  likely  require  the  target  to  be  within  a  30  -  40  cm  lateral  offset  (somewhat  site  dependent). 
Accordingly,  we  identified  all  cases  where  the  target  lateral  offset  was  greater  than  30  cm,  but 
the  corresponding  data  still  enabled  good  classification  of  the  target  (Figure  14). 
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Figure  14.  Example  of  good  classification  features  obtained  when  target  lateral  offset  is  greater  than  30  cm. 


After  completing  the  initial  analysis  and  subsequent  categorization  of  each  test  case,  we 
evaluated  the  effectiveness  of  different  data  and  model  parameters  in  separating  the  good 
classification  data  from  the  poor  classification  data.  Specifically,  we  searched  for  metrics  that 
would  produce  a  threshold  such  that  all  of  the  “poor  quality”  cases  could  be  placed  on  one  side 
the  threshold  while  as  many  “good  quality”  cases  as  possible  remained  on  the  other  side.  Figures 
15  and  16  show  examples  of  data  and  model  parameters,  respectively,  that  are  applied  as  quality 
metrics. 
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Figure  15.  Example  of  a  data  parameter  (in  this  case  data  noise)  used  as  a  quality  metric.  Each  dot  in  the  plot 
corresponds  to  a  specific  test  case.  Grey  dots  correspond  to  test  cases  that  produced  good  classification 
features;  red  dots  correspond  to  test  cases  that  produced  1  or  2  bad  polarizabilities  (poor  classification);  and 
blue  dots  correspond  to  test  cases  that  did  not  produce  any  good  classification  features.  The  threshold 
(dashed  line)  is  placed  so  that  all  poorly  classified  test  cases  (red  and  blue  dots)  are  on  one  side.  The  black 
arrow  indicates  the  direction  of  increasing  quality  (according  to  the  specific  metric  used).  Test  cases  in  which 
the  target  lateral  offset  exceeded  30  cm  are  also  marked  in  the  plot  with  squares. 

Figure  15  is  an  example  of  a  data  parameter  that  does  not  work  effectively  as  a  quality  metric.  In 
this  case  we  would  anticipate  that  decreasing  data  noise  might  correspond  to  increasing 
classification  quality;  however,  this  relationship  does  not  correlate  strongly.  Both  “good  quality” 
and  “poor  quality”  cases  are  distributed  fairly  evenly  across  this  parameter  space  and  when  the 
threshold  is  set  to  contain  all  “poor  quality”  cases,  almost  no  “good  quality”  cases  pass  the 
threshold. 
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Figure  16.  Example  of  a  model  parameter  (polarizability  uncertainty)  used  a  quality  metric.  Threshold 
setting  is  determined  by  “poor  quality”  outlier  (circled  in  blue). 

Figure  16  provides  an  example  of  a  model  parameter  applied  as  a  quality  metric.  In  this  case, 
polarizability  uncertainty  works  somewhat  effectively  as  a  quality  metric.  Most  of  the  “poor 
quality”  cases  (blue  and  red  dots  in  the  figure)  correspond  to  higher  uncertainty  values  while 
most  of  the  “good  quality”  cases  (grey  dots)  correspond  to  lower  uncertainty  values.  In  this 
analysis,  however,  a  threshold  setting  based  on  this  metric  does  not  work  very  well  because  of  a 
“poor  quality”  outlier.  Thus,  in  evaluating  quality  metrics  it  is  important  to  consider  not  only 
correlation  to  data  quality,  but  also  the  robustness  of  these  metrics.  Outliers  may  indicate  that  a 
particular  metric  may  not  be  robust  enough  to  use  for  in-field  quality  decisions. 

During  our  analysis  of  various  model  and  data  parameters,  we  found  several  that  showed 
promise  as  effective  quality  metrics;  however,  many  of  these  metrics  produced  some  degree  of 
outliers  in  the  parameter  space.  In  order  to  improve  upon  the  robustness  offered  by  individual 
quality  metrics,  we  started  creating  quality  indicators  that  were  based  on  several  different 
metrics.  These  indicators,  known  as  figures  of  merit,  provide  an  output  that  is  an  aggregation  of 
several  quality  metric  values.  Consequently,  a  figure  of  merit  can  be  much  more  robust  than  an 
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individual  parameter  that  is  used  as  a  quality  metric.  Figure  17  shows  an  example  of  an  effective 
figure  of  merit. 


Test  Case 


Figure  17.  Example  of  a  figure  of  merit  that  applies  several  quality  metrics.  This  figure  of  merit  shows  good 
correlation  to  data  quality  and  is  fairly  robust  in  that  it  does  not  produce  significant  outliers  in  the  feature 
space.  In  this  example,  the  figure  of  merit  could  effectively  improve  the  in-field  quality  decision  process  since 
there  are  several  cases  that  yield  good  classification  features,  but  correspond  to  large  lateral  offsets  (circled  in 
black).  These  are  instances  where  the  figure  of  merit  could  be  used  to  avert  recollects. 

The  figure  of  merit  presented  in  Figure  17  could  be  effective  for  improving  the  in-field  quality 
decision  process.  It  is  calculated  as  the  log  of  the  product  of  five  measures:  (1)  data  misfit 
(cumulative  point-to-point  absolute  difference  between  the  observed  and  predicted  data  scaled  by 
the  absolute  value  of  the  observed  data  amplitude);  (2)  correlation  between  observed  and 
predicted  data  (linear  relationship  between  the  predicted  and  observed  data;  does  not  account  for 
constant  offsets);  (3)  jitter  (time  series  point-to-point  difference)  in  the  observed  data;  (4) 
fraction  of  data  above  the  standard  deviation;  and  (5)  size  of  the  difference  between  L2  and  L3 
(secondary  and  tertiary  polarizabilities).  (1)  through  (3)  are  calculated  using  only  data  above  the 
standard  deviation.  (1)  through  (4)  are  data-based  measures;  (5)  is  a  model-based  measure.  Once 
the  threshold  is  set  to  contain  all  “poor  quality”  cases,  there  are  many  “good  quality”  cases  that 
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pass  the  threshold.  Many  of  these  “good  quality”  cases  correspond  to  targets  with  a  lateral  offset 
exceeding  30  cm.  Applying  the  figure  of  merit  to  the  quality  decision  process  would  avert 
recollection  of  these  data  sets,  thus  improving  the  overall  efficiency  of  the  survey. 

Table  1  shows  the  results  of  the  quality  metrics  analysis  and  compares  the  effectiveness  of 
individual  quality  metrics  to  the  aggregate  figure  of  merit.  The  threshold  value  in  Table  1 
corresponds  to  the  threshold  setting  that  provides  containment  of  all  “poor  quality”  cases  (i.e., 
the  operating  threshold).  The  percent  effective  value  is  the  percentage  of  “good  quality”  cases 
(out  of  the  total)  that  exceed  this  threshold.  This  percentage  is  an  indicator  of  how  well  the 
quality  metric  separates  good  data  from  bad  data.  It  is  evident  that  the  figure  of  merit 
outperforms  the  individual  metrics  significantly. 

Table  1 


Metric 

Threshold 

%Effective 

Model  SNR 

157.499 

9.8 

Data  Noise 

0.115 

1.1 

Figure  of  Failure 

0.264 

10.3 

Polarizability  Uncertainty 

1.067 

0.0 

LoglO(Polarizability  Jitter) 

0.288 

14.4 

LoglO(L23  Separation) 

-4 

0.0 

Correlation  Coefficient 

0.999 

1.1 

Figure  of  Merit 

0.456 

46.6 

Table  1  demonstrates  the  utility  of  applying  additional  metrics  to  support  data  quality  decisions 
that  would  otherwise  be  based  on  estimated  target  location  alone.  Applied  on  an  individual 
basis,  these  metrics  would  not  be  particularly  effective  for  separating  good  data  from  bad  data; 
however,  when  used  in  a  supporting  role,  they  could  significantly  reduce  the  necessity  for 
recollects.  This  additional  insight  would  be  particularly  important  during  projects  when  relying 
on  the  estimated  target  location  alone  would  lead  to  an  unnecessarily  high  percentage  of 
recollects. 

As  an  example,  consider  the  data  analysis  shown  in  Figure  17.  In  this  instance,  the  figure  of 
merit  effectively  identifies  46.6  percent  of  the  “good  quality”  cases.  Out  of  these  correctly 
identified  data,  approximately  23  percent  (19  data  files)  correspond  to  cases  where  the  target 
offset  exceeded  a  distance  of  30  cm  from  the  center  of  the  sensor.  Thus,  19  out  of  the  66  total 
“good  quality”  data  files  that  corresponded  to  offsets  exceeding  30  cm  were  correctly  identified 
using  the  figure  of  merit.  This  analysis  demonstrates  that  if  we  applied  the  figure  of  merit  to 
support  the  data  quality  decision,  we  would  achieve  an  almost  30  percent  (19/66)  reduction  in  the 
number  of  recollects  that  would  otherwise  have  been  required  if  we  had  relied  on  the  estimated 
target  location  metric  alone. 

3.3.  Implementing  Quality  Metrics 

Quality  metrics  provide  several  possibilities  for  enabling  sensor  operators  to  make  better  in-field 
quality  decisions.  Estimated  target  location  should  be  used  as  the  primary  metric  during  in-field 
QC.  Because  misguided  sensor  placement  (whether  a  result  of  operator  error  or  inaccurate  target 
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picking)  can  be  easily  rectified,  metrics  that  identify  sensor  positioning  errors  are  most  relevant 
to  in-field  QC.  Operators  can  use  the  estimated  target  location  to  identify  cases  where  sensor 
positioning  errors  are  likely  to  degrade  data  quality.  If  the  estimated  target  location  exceeds  an 
established  data  quality  threshold,  the  operator  may  immediately  reposition  the  sensor  and 
recollect  based  on  this  information. 

As  we  saw  from  our  previous  analysis  of  quality  metrics,  however,  relying  on  estimated  target 
location  alone  can  potentially  lead  to  unnecessary  recollects  (i.e.,  high  quality  data  files 
corresponding  to  target  offsets  that  exceed  the  pre-defined  data  quality  threshold). 
Consequently,  a  secondary  subset  of  data  and  model  parameters  recovered  from  in-field  analysis 
of  the  data  may  be  used  to  further  support  quality  decisions.  These  parameters  can  be  combined 
to  produce  a  figure  of  merit  that  provides  additional  insight  into  the  data  quality.  This 
information  should  be  used  to  identify  cases  where  the  estimated  location  exceeds  the  established 
threshold,  but  data  quality  remains  high. 

While  it  is  possible  to  apply  these  primary  and  secondary  quality  metrics  as  standard  practice,  the 
actual  thresholds  selected  for  these  metrics  will  most  often  depend  on  the  specific  site  where  they 
will  be  used.  The  estimated  location  threshold  should  be  determined  based  on  the  complexity  of 
the  site.  While  the  30  cm  threshold  provides  a  good  rule  of  thumb,  sites  that  contain  a  low 
density  distribution  of  anomalies  may  allow  for  larger  offset  thresholds.  Conversely,  high 
density  sites  or  areas  that  contain  magnetic  geology  are  more  complex  and  may  require  more 
conservative  thresholds.  A  careful  site  assessment  should  be  conducted  before  determining  the 
appropriate  offset  threshold. 

The  decision  to  use  other  supporting  quality  metrics,  or  a  figure  of  merit,  should  also  be 
influenced  by  the  site  complexity.  While  it  may  be  possible  to  reduce  the  number  of  recollects 
by  using  metrics  other  than  estimated  location,  the  reliability  of  these  supporting  metrics  may 
depend  on  the  site’s  target  density  and  geology.  The  best  practice  for  implementing  an 
additional  figure  of  merit  may  be  to  establish  its  reliability  using  an  Instrument  Verification  Strip 
(IVS)  and  calibration  procedure  prior  to  incorporating  it  during  a  production  survey. 

This  approach  is  similar  to  the  process  used  to  establish  Data  Quality  Objectives  (DQOs)  for 
detection  sensors.  As  an  example,  we  might  use  a  calibration  pit  to  establish  the  effectiveness  of 
a  figure  of  merit  for  specific  target  types.  Typically,  classification  library  data  files 
corresponding  to  seeded  targets  or  potential  targets  of  interest  would  be  acquired  in  a  calibration 
area  prior  to  the  start  of  cued  survey  operations.  Because  these  library  items  are  selected  based 
on  their  relevance  to  the  site,  they  would  provide  good  test  cases  for  establishing  figure  of  merit 
reliability.  Once  the  initial  reliability  of  supporting  metrics  is  established  for  the  relevant  target 
subset,  verification  could  be  conducted  in  the  IVS  by  acquiring  data  files  at  various  offsets  from 
each  IVS  item.  This  process  would  ensure  the  supporting  metrics  are  incorporated  in  the 
classification  DQOs. 

4.  System  Proveout 

After  completing  the  initial  development  of  the  real-time  QC  software,  we  performed  a  series  of 
tests  to  ensure  the  QC  module  has  obtained  the  planned  objectives: 
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1.  Inversion  of  static  MetalMapper  data  and  display  of  dipole-fit  parameters  within  1  second 
(nominal)  of  acquisition  completion; 

2.  Display  of  sensor  position  within  10  cm  of  true  GPS  coordinates; 

3.  Display  of  estimated  dipole/target  location  within  10  cm  of  true  anomaly  location. 

These  basic  capabilities  are  required  for  the  project  to  proceed  to  a  live  site  demonstration.  The 
following  subsections  present  the  results  of  our  testing  and  demonstrate  that  the  QC  module  has 
achieved  these  objectives. 

4.1.  Real-Time  Inversion 


The  file  watch  inversion  process  adds  minimal  time  to  the  existing  data  acquisition  cycle.  We 
performed  several  operational  tests  that  have  demonstrated  the  inversion  of  data  in  each  file 
typically  requires  0.2  -  0.4  seconds  to  complete  and  produce  results.  We  clocked  the  inversion 
time  for  many  sample  data  files  and  found  these  results  were  highly  repeatable.  Occasionally  the 
inversion  will  require  more  than  1  second;  however,  this  only  occurs  if  the  file  contains  data  that 
are  poorly  represented  by  the  model.  For  example,  when  we  perform  a  sensor  calibration  with  an 
object  placed  on  top  of  the  sensor,  the  inversion  time  is  slightly  longer.  Because  the  inversion 
constrains  the  target  space  to  a  region  below  the  sensor,  placing  an  object  above  the  sensor  will 
produce  poor  inversion  results.  It  is  very  rare  for  this  type  of  scenario  to  occur  during  an  actual 
cued  survey. 

The  additional  time  required  for  the  file  watch  inversion  is  negligible  when  compared  to  the 
typical  static  data  acquisition  period  of  approximately  20-30  seconds.  The  immediate  return  of 
dipole-fit  parameters  provides  the  operator  with  data  quality  metrics  while  the  MetalMapper  is 
still  in  proximity  to  the  target.  Once  the  proper  QC  module  flows  are  started  (steps  1  -  4  in 
subsection  2.1),  the  quality  metrics  (e.g.,  estimated  location,  model-fit  parameters,  etc.)  are 
automatically  displayed  in  the  navigation  and  QC  interfaces  once  the  acquisition  file  is 
generated.  Figure  18  shows  an  example  of  the  dipole-fit  results  displayed  in  these  interfaces. 


Estimated  Location 


Figure  18.  The  QC  module  updates  the  navigation  interface  (left)  and  QC  interface  (right)  with  estimated 
location  and  dipole-fit  parameters  immediately  after  a  MetalMapper  data  file  is  acquired  using  the  EM3D 
acquire  button  (green  button  on  bottom  interface  display). 
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4.2.  Sensor  Positioning  Accuracy 

To  verify  the  positioning  accuracy  of  the  QC  module  navigation  component,  we  conducted  a 
series  of  measurements  at  the  FLBGR  compound  test  pit.  We  started  this  process  by  surveying 
the  center  of  the  test  grid  with  an  RTK  differential  GPS  receiver  to  obtain  accurate  coordinates 
for  the  test  pit.  We  then  guided  the  MetalMapper  over  the  center  of  the  test  pit  using  a  set  of 
crosshairs  placed  directly  above  the  test  pit  center  (Figure  19).  We  estimated  that  this  method 
enabled  us  to  center  the  MetalMapper  over  the  test  pit  center  with  a  total  accuracy  of 
approximately  2-3  cm  (combined  GPS  and  manual  positioning  errors). 


Figure  19.  We  carefully  aligned  a  set  of  crosshairs  with  the  center  of  the  test  grid  (left)  and  then  manually 
guided  the  MetalMapper  sensor  head  (center,  right). 

Using  the  surveyed  coordinates  for  the  test  grid  center,  we  created  a  target  file  to  load  into  the 
QC  module  navigation  interface.  By  displaying  these  coordinates  in  the  navigation  interface,  we 
were  able  to  verify  that  the  module  provide  the  desired  accuracy  of  within  10  cm  of  the  ground 
truth  (Figure  20).  This  test  ensured  that  the  MetalMapper  GPS  and  IMU  were  properly 
calibrated  and  that  the  QC  module  interpreted  the  navigation  data  streams  correctly. 


Figure  20.  Navigation  display  showing  the  location  of  the  test  pit  center  (red  dot).  Positioning  of  the  sensor 
was  achieved  by  manually  guiding  the  MetalMapper  sensor  head  over  the  test  pit  center.  The  total 
positioning  error  for  this  process  was  estimated  to  be  2  -  3  cm.  It  is  evident  that  the  QC  module  provides  a 
positioning  accuracy  that  is  well  within  the  desired  10  cm  threshold. 
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4.3.  Estimated  Location  Reliability 

To  verify  the  reliability  of  the  estimated  location  metric,  we  collected  several  MetalMapper  data 
files  over  targets  placed  on  the  test  grid.  We  varied  the  lateral  offset  from  the  center  of  the  grid 
for  each  test  case  to  provide  a  range  of  locations.  We  then  compared  the  estimated  target 
location  to  the  actual  ground  truth  location  (based  on  the  grid  coordinates)  to  determine  the 
accuracy  of  this  metric.  We  found  that  the  estimated  location  was  well  within  the  desired  10  cm 
threshold.  In  fact,  this  accuracy  was  maintained  even  at  large  lateral  offsets  where  the  targets 
were  placed  near  the  comers  of  the  grid  (~70  cm  offset).  Figure  21  shows  an  example  of  the 
estimated  location  accuracy. 


Figure  21.  Estimated  location  verification  tests.  Targets  were  placed  at  several  locations  on  the  test  grid 
(right)  and  the  QC  module  was  used  to  estimate  the  target  coordinates  (left).  Tests  verified  the  10  cm 
accuracy  was  achieved. 

4.4.  Software  Functionality 

We  tested  the  overall  functionality  of  the  QC  module  by  performing  a  mock  cued  survey.  We 
acquired  GPS  coordinates  for  several  locations  within  the  FLBGR  compound  area  and  placed 
target  objects  within  a  50  cm  radius  of  each  location  (the  offset  was  varied  to  establish  realistic 
target  pick  errors).  We  loaded  these  target  pick  coordinates  into  the  QC  module  (step  2,  section 
2.1)  and  conducted  a  cued  survey  at  each  location.  We  tested  the  software  functionality  by  going 
through  the  process  of  failing  an  initial  acquisition,  repositioning  the  sensor  using  the  navigation 
interface  and  the  estimated  target  location,  and  recollecting  in  the  new  location.  An  example  of 
this  sequence  follows  (Figures  22  -  32): 
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Figure  22.  Mock  cued  survey  performed  at  FLBGR  compound.  Target  objects  were  placed  near  surveyed 
locations  around  the  site. 


Figure  23.  Navigation  interface  showing  sensor  position  over  one  of  the  target  pick  coordinates  (red  dot).  The 
EM3D  software  is  in  the  process  of  acquiring  a  static  data  measurement. 
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Figure  24.  Once  the  acquisition  is  complete  and  the  data  file  is  created,  the  QC  module  updates  the  navigation 
interface  with  the  estimated  target  location  (pink  dot). 


Figure  25.  The  QC  interface  is  also  updated  to  display  the  recovered  location,  model  parameters,  and  best 
library  match  (note:  some  of  these  parameters  do  not  display  properly  here  due  to  the  resolution  of  the 
monitor  on  which  this  image  was  captured). 
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Figure  26.  The  QC  interface  prompts  the  operator  to  assess  the  quality  of  the  acquisition  based  on  the 
estimated  location  and  model  parameters.  This  particular  case  will  be  selected  as  a  “fail”  due  to  the  large 
target  offset  indicated  by  the  estimated  target  location. 
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Figure  27.  Once  the  initial  acquisition  is  selected  as  a  “fail”,  the  navigation  interface  is  updated  to  reflect  this 
decision.  The  initial  pick  coordinates  are  changed  to  grey  and  the  estimated  location  coordinates  are  changed 
to  red.  The  operator  now  repositions  the  sensor  based  on  these  new  coordinates  (red  dot). 
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Figure  28.  Once  the  operator  repositions  the  sensor  over  the  estimated  target  location  coordinates  (red  dot), 
the  operator  can  recollect  the  data. 


Figure  29.  After  the  recollect  is  complete,  the  QC  interface  is  updated  with  the  recollect  inversion  results.  In 
this  example,  the  estimated  target  location  (pink  dot  in  the  position  plot)  is  now  very  close  to  the  center  of  the 
sensor. 
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Figure  30.  The  QC  interface  once  again  prompts  the  operator  to  make  a  quality  decision  based  on  the  new 
inversion  results.  This  case  will  now  be  selected  as  a  “pass”  due  to  the  consistent  estimated  target  location. 
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Figure  31.  When  the  recollect  is  accepted,  the  estimated  target  location  coordinates  are  changed  to  green. 
The  original  target  pick  coordinates  remain  grey.  The  operator  can  now  proceed  to  the  next  target  pick. 


Figure  32.  The  actual  target  (37  mm)  and  the  initial  target  pick  coordinates  (orange  mark)  corresponding  to 
the  survey  described  in  the  last  several  figures. 
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By  verifying  the  functionality  of  the  QC  module  during  a  mock  survey,  we  were  able  to  assess 
how  well  these  additional  in-field  QC  steps  can  be  incorporated  in  a  cued  survey,  and  whether 
they  would  add  any  complexity  to  the  overall  survey  process.  We  repeated  the  mock  survey  for 
numerous  target  picks  and  allowed  several  different  operators  to  try  the  QC  routine.  Our  initial 
assessment  based  on  these  tests  is  that  the  QC  module  adds  a  very  intuitive  sequence  to  the  cued 
survey  procedure.  Once  the  QC  and  navigation  flows  are  initialized,  and  the  EM3D  parameters 
are  selected,  very  few  additional  steps  are  required  during  the  actual  survey.  The  general 
consensus,  based  on  feedback  from  several  operators,  was  that  the  QC  module  provides 
sufficient  information  for  enabling  quality  decisions  and  corrective  action,  but  not  so  much 
information  that  interpretation  of  the  results  becomes  onerous. 

5.  Demonstration  Phase 

Following  the  proveout  phase  of  this  project,  we  will  further  evaluate  our  in-field  QC  process 
through  a  live  site  demonstration.  While  the  proveout  phase  allowed  us  to  assess  the  basic 
functionality  of  the  software  module  and  verify  that  it  provides  the  desired  capabilities,  a  live  site 
demonstration  will  provide  more  insight  into  the  operational  aspects  of  implementing  an  in-field 
QC  component  to  cued  surveys. 

As  our  overarching  objective  is  to  improve  the  efficiency  of  production  surveys  by  ensuring  in¬ 
field  that  cued  data  are  acquired  at  a  quality  sufficient  for  classification,  we  will  evaluate  the 
effectiveness  of  our  approach  using  two  principal  performance  metrics:  (1)  total  anomaly 
acquisition  rate;  and  (2)  total  percentage  of  recollects. 

To  implement  these  metrics,  we  will  demonstrate  the  in-field  QC  process  at  a  production-level 
site  and  compare  the  performance  of  our  approach  against  that  of  the  current  practice  (i.e.,  off¬ 
site  quality  analysis  only).  We  will  perform  a  static  MetalMapper  survey  of  a  statistically 
relevant  number  of  samples  (-1000-1500)  while  incorporating  our  in-field  QC  module.  The 
number  of  acquisitions  requiring  corrective  action  (i.e.,  a  recollect)  will  indicate  the  number  of 
anomalies  that  would  require  a  resurvey  (i.e.,  a  reacquire)  if  the  in-field  quality  process  were  not 

in  place.  As  part  of  this  demonstration,  we  will  implement  the  quality  metrics  analysis 

conducted  as  part  of  the  initial  proveout  phase  to  determine  the  necessity  for  data  recollects 
during  the  survey.  Based  on  the  specific  site  characteristics  (e.g.,  anomaly  density,  geology, 
etc.),  we  will  define  the  quality  objectives  for  each  metric.  For  example,  the  estimated  location 
offset  threshold  will  be  chosen  to  reflect  the  targets  of  interest,  the  depth  of  detection,  and  the 
local  anomaly  density.  Additionally,  the  site  characteristics  will  also  determine  how  we  apply 
additional  metrics  in  the  form  of  a  figure  of  merit.  It  may  be  beneficial,  for  this  demonstration, 
to  apply  a  figure  of  merit  during  post  analysis  to  determine  how  many  recollects  could  have  been 
averted. 

To  assess  the  overall  efficiency  gains,  the  acquisition  rate  for  anomalies  that  do  not  require 
recollects  will  indicate  the  baseline  acquisition  rate  for  the  current  practice  (the  real-time 

inversion  adds  a  negligible  amount  of  time  to  each  acquisition).  For  acquisitions  requiring 

recollects,  we  will  compare  the  timestamp  of  the  initial  data  acquisition  to  that  of  the  corrective 
data  acquisition.  This  difference  will  indicate  the  additional  time  required  to  perform  a  recollect. 
To  determine  the  overall  reduction  in  survey  time  afforded  by  the  real-time  quality  control 
process,  we  will  compare  the  total  time  spent  on  recollects  to  the  total  time  that  would  be 
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required  to  redeploy  and  reacquire  those  anomalies  at  the  baseline  acquisition  rate.  Thus,  the 
outcome  of  this  demonstration  will  be  a  quantifiable  assessment  of  any  productivity  gains 
resulting  from  the  in-field  quality  analysis. 

6.  Future  Development 

Based  on  our  testing  to  date,  the  QC  module  shows  significant  promise  for  improving  the 
efficiency  and  overall  quality  of  production  cued  EMI  surveys.  The  ability  to  perform  real-time 
inversions  of  the  cued  sensor  data  and  present  the  corresponding  quality  metrics  to  sensor 
operators  will  enable  in-field  quality  decisions  that  will  likely  produce  better  classification 
results. 

Incorporating  the  QC  module  into  live  site  demonstrations  and  production  MMRP  surveys  will 
help  to  establish  the  best  practices  for  implementing  these  quality  metrics.  For  example, 
production  surveys  will  require  DQO’s  to  establish  the  reliability  of  these  metrics  at  each  site. 
The  more  opportunities  that  we  have  to  practice  in-field  quality  decisions,  the  more  information 
we  will  acquire  to  guide  the  development  of  these  DQO’s  and  quality  metric  thresholds. 

As  we  look  beyond  the  current  scope  of  this  project,  there  are  several  opportunities  to  further 
extend  in-field  QC  capabilities  based  on  the  initial  development  conducted  during  the  proveout 
phase.  Specifically,  these  opportunities  include: 

1.  Incorporation  of  multi-object  inversion  capabilities  into  the  in-field  software  module. 
This  would  enable  better  quality  decisions  in  high  density  areas.  Ideally,  adding  the 
multi-object  capability  would  be  transparent  to  the  operator;  the  decision  to  use  a  multi¬ 
object  inversion  versus  a  single  object  inversion  would  be  automated  and  dependent  on 
the  value  of  certain  model  parameters. 

2.  Extension  of  the  in-field  QC  module  to  additional  cued  EMI  sensors  such  as  the 
TEMTADS  and  MPV  instruments.  The  MetalMapper  shows  significant  promise  for 
gaining  MMRP  acceptance;  however,  other  sensors  are  likely  to  follow.  By  applying  the 
QC  module  to  other  sensors  that  implement  different  operational  modes  (e.g.,  handheld, 
towed,  etc.),  we  can  ensure  that  the  quality  practices  are  developed  for  these  instruments 
before  they  are  incorporated  in  MMRP  projects. 

We  are  confident  that  the  QC  module  provides  capabilities  that  are  highly  relevant  to  current 
MMRP  practices.  Further  demonstration  of  these  capabilities  will  enable  us  to  refine  in-field  QC 
practices  to  ensure  the  future  success  of  classification  surveys  performed  in  production 
environments. 
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